
Calhoun 

iniQiuiic^iul Ar{hiv« of tilt Mil vdl Poii^roduiit School 


Calhoun: The NPS Institutional Archive 
□Space Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


2006-12 

A critical analysis of the Acquisition Review 
Journal: are we in step with the field? 

Miranda, Cristina M.; Spann, Cheronda V. 

Monterey, California. Naval Postgraduate School 


http://hdl.handle.net/10945/10092 


Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


htt p://w ww. n ps. e du/l ib ra ry 


Caflwuo is the Naval Postgraduate School's public access digital repository for 
research mate rials and institutiional putilicatiaos created by the NPS community. 
Calhoun is named for Professor of Mathematics Guy K. Caftiouo, NPS's first 
appointed — and putJlished — schoteily author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 Univefsity Circle 
Monterey, California USA 93943 







NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY, CALIFORNIA 


MBA PROFESSIONAL REPORT 


A Critical Analysis of the Acquisition Review Journal: 
Are We in Step with the Field? 


By: Cristina M. Miranda 

Cheronda V. Spann 

December 2006 

Advisors: Bryan Hudgens 

Keith Snider 


Approved for public release; distribution is unlimited. 






THIS PAGE INTENTIONALLY LEET BLANK 



REPORT DOCUMENTATION PAGE 


Form Approved 0MB No. 0704-0188 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for 
reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and 
reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of 
information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information 
Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of 
Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 

2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 
15 December 2006 MBA Professional Report 

4. TITLE AND SUBTITLE: A Critical Analysis of the Acquisition Review 5. LENDING NUMBERS 

Journal: Are We in Step with the Field? 

6. AUTHOR(S) Cristina M. Miranda and Cheronda V. Spann 

7. PEREORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PEREORMING 

Naval Postgraduate School ORGANIZATION REPORT 

Monterey, CA 93943-5000 _ NUMBER _ 

9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING / MONITORING 
N/A AGENCY REPORT NUMBER 

II. SUPPLEMENTARY NOTES The views expressed in this report are those of the author(s) and do not reflect the official 
policy or position of the Department of Defense or the U.S. Government. 


13. ABSTRACT (maximum 200 words) 

The purpose of this study is to provide contributing authors with an understanding of the trends in article submission to the 
Acquisition Review Journal (ARJ) with regard to the types of research performed, research design, and data analysis. This 
research will provide future contributors with insight that will improve both the quality of the ARJ and future research for the 
Acquisition Corps. It will also provide guidance and recommendations for future research articles within the ARJ. 

This study analyzed and classified 233 articles that were published in the ARJ over the last 13 years (1994-2006). Content 
and statistical analyses were performed on the themes, research types, research designs, and data analysis methods employed. 
Moreover, trends such as educational and institutional affiliations of contributing authors were also reviewed. 

The ARJ has shown some distinctive trends, which are reflected in its publication of a number of qualitative studies; 
however, it has also shown progress in the number of published quantitative studies. The academe and practitioners’ 
contributions remained steady, while civilian contributions have been rising. These trends are in line with current 
recommendations of the research community. For a relatively new journal, such trends are a good representation of a 
growing community. 


16. PRICE CODE 


NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) 

Prescribed by ANSI Std. 239-18 


20. LIMITATION 
OE ABSTRACT 

UL 


15. NUMBER OE 
PAGES 

103 


14. SUBJECT TERMS Acquisition, Acquisition Reform, Acquisition Reform Initiatives, Acquisition 
Review Journal, Content Analysis, Types of Research, Categorical Analysis, Research Designs, Types 
of Data Analysis 

18. SECURITY 
CLASSIEICATION OE THI S 
PAGE 

Unclassified 


19. SECURITY 
CLASSIEICATION OE 
ABSTRACT 

Unclassified 


17. SECURITY 
CLASSIEICATION OE 
REPORT 

Unclassified 


I2b. DISTRIBUTION CODE 

A 


I2a. DISTRIBUTION / AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited. 


I. AGENCY USE ONLY (Leave blank) 


1 




























THIS PAGE INTENTIONALLY LEET BLANK 


11 



Approved for public release; distribution is unlimited. 


A CRITICAL ANALYSIS OF THE ACQUISITION REVIEW JOURNAL: 
ARE WE IN STEP WITH THE FIELD? 


Cristina M. Miranda, First Lieutenant, United States Air Force 
Cheronda V. Spann, First Lieutenant, United States Air Force 


Submitted in partial fulfillment of the requirements for the degree of 


MASTER OF BUSINESS ADMINISTRATION 

from the 

NAVAL POSTGRADUATE SCHOOL 
December 2006 


Authors: _ 

Cristina M. Miranda 


Cheronda V. Spann 


Approved by:_ 

Bryan Hudgens, Lead Advisor 


Keith Snider, Support Advisor 


Robert N. Beck, Dean 

Graduate School of Business and Public Policy 



THIS PAGE INTENTIONALLY LEET BLANK 


IV 



ABSTRACT 


The purpose of this study is to provide contributing authors with an understanding 
of the trends in article submission to the Acquisition Review Journal (ARJ) with regard to 
the types of research performed, research design, and data analysis. This research will 
provide future contributors with insight that will improve both the quality of the ARJ and 
future research for the Acquisition Corps. It will also provide guidance and 
recommendations for future research articles within the ARJ. 

This study analyzed and classified 233 articles that were published in the ARJ 
over the last 13 years (1994-2006). Content and statistical analyses were performed on 
the themes, research types, research designs, and data analysis methods employed. 
Moreover, trends such as educational and institutional affiliations of contributing authors 
were also reviewed. 

The ARJ has shown some distinctive trends, which are reflected in its publication 
of a number of qualitative studies; however, it has also shown progress in the number of 
published quantitative studies. The academe and practitioners’ contributions remained 
steady, while civilian contributions have been rising. These trends are in line with 
current recommendations of the research community. For a relatively new journal, such 
trends are a good representation of a growing community. 
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I. 


INTRODUCTION 


A. BACKGROUND 

1. History of Acquisition Reform 

In 1961, Secretary of Defense Robert S. McNamara attempted to revamp the 
procurement systemd From 1961 on, several acquisition reform initiatives followed as 
shown in Figure 1. 


Year 

Initiative 

1961 

McNamara initiatives 

1970 

Fitzhugh Commission 

1972 

Commission on Government Procurement 

1976 

0MB Circular A-109 

1978 

Defense Science Board Acquisition Cycle Study 

1979 

Defense Resources Management Study 

1981 

Defense Acquisition Improvement Program 

1983 

Grace Commission 

1986 

Packard Commission 

1986 

Goldwater Nichols 

1989 

Defense Management Review 


Figure 1. Reform Initiatives^ 

These reform efforts emphasized streamlining the weapons acquisition process, 
improving cost estimating practices, and changing personnel procedures to produce more 
qualified contracting staff. Recommendations have included: 

• eliminating needless legal encumbrances on contracting procedures; 

• empowering program managers; 

• establishing clear lines of authority; 

• simplifying the source selection process; 


^ Deborah Frank, “A Theoretical Consideration of Acquisition Reform,” Acquisition Review 
Quarterly, Summer 1997, p. 281. 

2 Ibid. 
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• reducing technical criteria; 

• re-codifying federal laws governing procurement; 

• employing more frequent product testing and competitive prototyping; 

• improving the pay, training, and career options for personnel; and 

• multi-year congressional funding.3 

The efforts mentioned above are themes that have been expressed throughout the 
articles and issues of the Acquisition Review Journal (ARJ). The ARJ was intended for 
the following purpose: 

. . . Acquisition Corps members and other readers from 
government. Congress, industry and academe are encouraged to use the 
Acquisition Review Quarterly (ARQ), now ARJ, as their professional 
forum for discussion and exchange of policies, research, information, 
and opinion.4 

2. Relevance of the ARJ 

The ARJ encourages articles on policy, scholarly research, opinions, and anything 
else that affects the acquisition community. Acquisition is a very broad term, which is 
why the ARJ welcomes articles in all areas such as contracting, logistics, management, 
and technology. For example, in February 2006, Lieutenant General Donald J. Hoffman, 
United States Air Force (Military Deputy, Office of the Assistant Secretary of the 
Air Force for Acquisition) stated the following: “I stand by my earlier comments on the 
quality of our military hardware, but you have all read the headlines about acquisition 
reform and the biggest driver is the need to control cost and schedule.”3 His viewpoint is 
reflected in this study’s results as Cost, Schedule, and Performance is the second highest 
theme category of the ARJ, with Technology being number one. 


3 Lauren Holland, “The Weapons Acquisition Process; The Impediments to Radical Reform,” 
Acquisition Review Quarterly, Spring 1998, p. 235. 

‘^Acquisition Review Quarterly, Editorial Mission, Summer 1994, no page number noted. 

3 Lieutenant General Donald J. Hoffman, United States Air Force, “The Way Ahead,” February 2006, 
httt)s://www.safaq.hq.af.mil/news/februarv2006/hoffman.html . last accessed on 29 November 2006. 
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B. HISTORY OF THE ARJ 


One of the two publications by the Defense Acquisition University (DAU), aside 
from the Acquisition, Technology, and Logistics (AT&L) magazine, is the ARJ, which has 
consistently published articles for both academe and practitioners with a particular 
interest in the acquisition community. During its 13-year life span, it has published 
42 issues and 233 articles, not including editorial and special edition introductions and 
books reviews. A total of 325 authors have published articles in the ARJ (see 
Appendix A). The ARQ was renamed the ARJ for Issue 35 (2004), concurrent with a 
decrease in the number of issues published per year from four to three. Although this 
publication experienced a name change, the content, intent, and mission did not change. 
The ARJ was designed to address the needs of professionals across the defense 
acquisition spectrum. Its editorial mission is to: 

. . . provide practicing acquisition professionals with relevant 
management tools and information based on recent advances in policy, 
management theory, and research. The ARQ (ARJ) addresses the needs of 
professionals across the full spectrum of defense acquisition, and is intended 
to serve as the mechanism for fostering and disseminating scholarly research 
on acquisition issues, for exchanging opinions, for communicating policy 
decisions, and for maintaining a high level of awareness regarding 
acquisition management philosophies. In addition to the acquisition 
professional, the ARQ (ARJ) provides insight to others in the Department of 
Defense (DoD), Congress, industry and academe who have significant 
interest in how the DoD conducts its acquisition mission. Acquisition Corps 
members and other readers from government. Congress, industry and 
academe are encouraged to use the ARQ (ARJ) as their professional forum 
for discussion and exchange of policies, research information, 
and opinions.6 

Based on the editorial mission statement, the ARJ should serve as a forum for the 
exchange of opinions, communicating policy decisions, and for maintaining awareness 
for acquisition management philosophies. Not only should it be a forum for the 
community, but it should also be used as a means for communication, both for the 
members of the acquisition community, and to all those that have an expressed interest in 


^Acquisition Review Quarterly, Editorial Mission, Summer 1994, no page number noted. 
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the DoD’s acquisition mission.^ Members of the Acquisition Corps and other 
communities (e.g., logistics, contracting, end user, etc.) are encouraged to use the ARJ as 
a professional platform for information exchange among community members. 
Gerald E. Knightley, the Executive Director at the time of its inception, believes that the 
ARTs target audience is senior members of the Acquisition Corps.^ 

The ARJ has had four editors over its 13-year existence (see Table 1). In Table 1, 
Norene Taylor is referred to as an editor; however, in Issue 32 (published in summer 
2003) the position of editor was disseminated among three positions and she was 
designated as Managing Editor (lead editor). 


Time Period 

Issues 

Editor 

Winter 1994-Spring 1995 

5 

Robert W. Ball 

Summer 1995-Eall 1997 

6 

James Kurt Wittmeyer 

Winter 1997-Summer 2000 

10 

Deborah E. Gonzalez 

Eall 2000-Present (2006) 

21 

Norene E. Taylor 

(Formally Blanch and Fagan-Blanch) 


Table 1. Ai?7 Editors 


At the end of each issue of the ARJ, there is a section called “Guidelines for 
Authors,” which specifies the ARJ's criteria for manuscript submission. In the 1994 
winter edition, for example, the ARJ was interested in articles that 

. . . represent scholarly examination, disciplined research, and 
supported empirical experience in the fields of defense systems 
management and acquisition management. . . . Manuscripts supporting the 
Defense Acquisition University (DAU) commitment to improve the 
acquisition process and the professionalism of the acquisition workforce 
are particularly welcome.^ 

While articles related to acquisition were preferred, other fields of management 
would not be omitted. 


^ Gerard E. Knightley, “Acquisition Review Quarterly,” Memorandum for Defense Acquisition 
Professionals, Acquisition Review Quarterly, Winter 1994. 

8 Ibid. 

9 Acquisition Review Journal, “Guidelines for Authors,” Winter 1994, p. 84. 
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The summer 1995 issue experienced a cover change, but it also had an addendum 
to the “Guidelines for Author” section in regard to the types of manuscripts it desired to 
publish: “The ARJ welcomes manuscripts from anyone interested in the defense 
acquisition process.Further change occurred in the winter 1997 issue, which 
explicitly defined defense acquisition “. . . as the conceptualization, initiation, design, 
development, logistics support, modification, and disposal of weapons and other systems, 
supplies, or services to satisfy Defense Department needs, or intended for use in military 
missions.”! 1 The next noticeable change was in the summer 1999 issue, in which authors 
were asked to include a photograph of themselves and their email address, to be placed 
with their biography. In Issue 37, in the “Guidelines for Authors” section, the ARJ began 
to publish planned issue themes and publication schedules. In this section, they displayed 
the due date and publication date; however, in order to learn the themes, prospective 
authors needed to view the DAU’s Website (http://www.dau.mil). Another change, made 
in Issue 39, added themes to the “Guidelines for Authors” section, right under the due 
date and publication date. 

Over the years, the ARJ evolved in order to better serve the acquisition 
community. Chapter II begins to explore the trends, categories, and themes that were 
gathered from reading the articles that have been published since its inception. 

C. PURPOSE OF STUDY 

1. Research Statement 

Prior to 2005, a study of the ARJ had not been conducted. However, in 2005, 
Elder did a thesis on the ARJ. In his study, he reviewed all the articles published in the 
ARJ up to Issue 36 (released in 2004), which provided him with a total of 193 articles 
after he excluded editorial and special edition introductions and book reviews. This study 
will include the articles that Elder reviewed plus the remaining ones to date, providing us 
with a grand total of 233 articles. 


^^Acquisition Review Journal, “Guidelines for Authors,” Summer 1995, back cover. 
!! Acquisition Review Journal, “Guidelines for Authors,” Winter 1997, p. 127. 
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The purpose of this study is to provide contributing authors with an understanding 
of the trends in article submission to the Acquisition Review Journal (ARJ) with regard to 
the types of research performed, research design, and data analysis. This research will 
provide future contributors with insight that will improve both the quality of the ARJ and 
future research for the Acquisition Corps. It will also provide guidance and 
recommendations for future research articles within the ARJ. 

The intentions of this study are: (1) to replicate Elder’s thesis and compare the 
two sets of results; (2) to analyze the remaining articles to date and determine if there has 
been any noticeable changes in trends in the types of articles, the themes, and the 
characteristics of the contributing authors; and (3) to analyze these trends over the past 
13 years within the acquisition community. Replicating Elder’s thesis and conducting a 
second content analysis of the ARJ will provide confirmation of its focus on Acquisition 
Reform and if it truly provides a forum for the acquisition community as described in the 
ARJ mission. 

Other research communities have performed studies on research that has been 
done in their respective communities. An example of this type of study was performed 
by two Supply Chain Management professors, Craig R. Carter and Eisa M. Ellram, who 
published the article, “Thirty-Eive Years of the Journal of Supply Chain Management: 
Where Have We Been and Where are We Going?” In their study, they reviewed all the 
articles within the Journal of Supply Chain Management and noted the 
following information: 

• The editors over the existence of the journal 

• The authors 

o Their educational background 

o Their institutional affiliation 

• The article’s subject categories 

• The type of research performed 

• The type of research design 

• The type of data analysis 
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Once all the information was collected, they completed a content analysis of the 
data and noted any trends that existed. Their research was well designed and so 
comprehensive that the Acquisition Community felt it would be both interesting and 
beneficial to conduct a similar study on the ARJ. 

This study is a replication of the content analysis of the ARJ using two coders 
(researchers). The two coders have the same professional background and educational 
experience. The present endeavor compares the results of the two independent studies. 
In addition, the two coders separately analyzed the articles published in the ARJ through 
2006. A third coder was used as an arbitrator to resolve any differences in analysis. 

2. Research Questions 

Having read the ARJ's editorial mission, the following questions were structured. 

• How does the ARJ contribute to the acquisition community as a forum for 
intellectual exchange? 

• Who is publishing the articles in the ARJl 

• What determines the caliber of the articles published and why are those 

articles chosen over others? 

3. Investigative Questions 

Based on Carter and Ellram’s content analysis methodologies, the coders 
developed similar research questions to guide their primary research. 

• What subject matters or themes have been addressed in the ARJl 

o Do the subject matters or themes enable the readers to gain new 
insight about a particular acquisition issue; develop new concepts 
or theoretical perspectives about acquisition; or discover problems 
and/or solutions to particular issues? 
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• How have the themes been explored or written in the past 13 years? 

o What research methodologies have been employed in the journal? 
What is the most commonly employed research methodology? 

o What type of data collection and data analysis did the authors use 
to support their research/contribution? From among the different 
types of data analysis, which appears to be most common? 

o Are there trends in the type of articles, research categories, or data 
analysis over the years? Do the types differ across time? 

o What does this analysis suggest about the study of acquisition 
reform in the ARJl 

• What types of authors have contributed articles to the ARJl 

o Are they academe or practitioners? 

o What are their institutional affiliations? Do the authors come from 
military or civilian institutions? 

o What is the trend on the number of authors per article over the 
years? Did the number of authors per article change across time? 
Are the authors encouraged to collaborate with other authors? 

o Do the articles published to this point appear relevant with 
on-going acquisition reform? 

D. ORGANIZATION OF RESEARCH 

The study is organized into four chapters. Chapter I provides an introduction and 
overview of the study. Chapter II describes the methodology, including the processes 
and techniques used to acquire and analyze the data. Chapter III provides an in-depth 
analysis of the data, including the results from the replication of Elder’s study. Finally, 
Chapter IV summarizes the conclusions and provides recommendations for future study. 
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II. RESEARCH METHODOLOGY 


The study focused on classifying 233 articles that were published in the ARJ over 
the last 13 years. The coders read all the articles published since the journal’s inception 
in 1994, with the exception of editorial and special edition introductions and book 
reviews. In this chapter, we will discuss the methodology used to classify the articles in 
order to discuss different aspects of the ARJ. The methodology includes types of 
research performed, types of research design, and types of data analysis, statistical 
methods, and themes. 

A. DATA COLLECTION 

To gather all the data required for the analysis, the coders first created an Excel 
spreadsheet to capture all the areas relevant to achieving the answers to the research 
questions. The coders performed a content analysis of the themes, research types, 
research designs, and data analysis methods employed. A number of other authors have 
conducted studies of this stature within their respective communities such as: 

• Carter and Ellram 

• Das and Hanfield 

• Williams and Oumlil 

• Mentzer and Kahn 

The study performed a content analysis of the themes, research types, research 
designs, and data analysis methods employed. 

In order to collect the data, the process required both coders to read articles 
independently, while simultaneously filling in two data spreadsheets. Upon completion 
of the articles published in 1994 and 1995, a meeting of the coders was held to compare 
results. The reason for the meeting was to ensure that both individuals had similar 
interpretations of the definitions they applied to the articles, respectively. Each article 
was discussed individually and an exchange of the themes and categories took place. The 
articles on which the coders were in 100 percent agreement were noted and upon 
completion of all 233 articles, a reliability rate was calculated. The reliability rate 

9 



consists of the total number of articles that were in 100 percent agreement divided by the 
total number of articles (233), which resulted in a 91 percent agreement rate. This 
percentage is known as the inter-coder agreement, in which an agreement rate of 
70 percenti2 or higher is favorable. For those articles about which there was not 
100 percent agreement, a third coder was used to settle any discrepancies. The purpose 
of the third coder was to read the articles and perform the same assessment, which was 
used to adjudicate any disagreements. For instance, if the first coder assigned a 
Normative review, while the second assigned a Literature review, the disagreement was 
broken by the categorization of the third coder. The importance of the third coder was to 
ensure that a one-to-one ratio was maintained for each category per article. 

In the beginning, the coder meetings were critically important since the 
definitions of the categories were being refined for proper interpretation by both 
individuals. According to Rust and Cooil, reliability can be improved in many ways. 
One good option is to add more coders, but this option is not always feasible. Another 
option, to make it more practical and to improve the quality of the judging itself, is to 
make the instructions clearer and/or improve the methodology. Additionally, better 
reliability can be achieved through inter-coder agreement; here the coders provide 
information and rationale rather than random guesses. ^3 The coders focused on following 
a clear methodology, which led to high inter-coder agreement. 

In order to perform the categorical and statistical analyses, both researchers were 
responsible for counting the number of categories chosen for each research type within 
their own spreadsheet. Each coder used a different method to conduct a count of the 
categories: one coder used Excel to sort and count (histogram and count-if functions), 
while the other coder counted manually. The two methods derived the exact same 
results. Afterward, the two coders compared their count results to see if they were both 
in agreement, then they calculated the frequencies for each of the three types. 


John C. Nunnally and Ira Bernstein, Psychometric Theory, McGraw Hill, 1994, p. 265. 

13 Roland T. Rust and Bruce Cooil, “Reliability Measures for Qualitative Data: Theory and 
Implications,” Journal of Marketing, February 1994, p. 2. 
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B. TYPE OF RESEARCH PERFORMED 

The type of research performed was one of the three areas identified for each of 
the 233 articles. In order to classify each article, the coders used Mentzer and Khan’s 
topology. xhe types and the definitions that were used to categorize each article are 
displayed in Table 2. The frequency column in Tables 2, 3, and 4 displays the percentage 
of times that the respective category was chosen among the 233 articles. The categories 
were noted and tabulated, after all disagreements were settled, and then the total number 
of articles per category was divided by 233 (the total number of articles). The frequency 
tells the community the number of times that a particular category has been used in 
the ARJ. 

The agreement rate between the two coders was 96 percent—considerably above 
the 70 percent minimum inter-coder agreement rate and better than the 85 percent 
coefficient of agreements!^ recommended by Harold Kassarjian in 1977. As mentioned 
in Section A, a third coder and meetings were used to adjudicate any disagreements. 


14 John Mentzer and Kenneth A. Khan, “A Framework of Logistics Research,” Journal of Business 
Logistics, Vol. 16, No. 1, 1995, p. 241. 

1^ Harold H. Kassarjian, “Content Analysis in Consumer Research,” Journal of Consumer Research, 
Vol. 4, No. 1, 1997, pp. 8-18. 
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Category 

Definition 

Frequency 

Normative 

Review 

“Research that examines what ought to be and what individuals 
and organizations ought to do.”i6 Literature might be cited in 
the article, but the point of the inclusion of this literature is to 
support the opinions/assertions of the author. Normative 
research aims at improvements, which means that it includes 
evaluation of the present state of things and also of the 
direction of future development. 

28.3% 

Literature 

Review 

This type is a review and synthesis of existing literature, the 
result of which is the development of a framework, 
propositions, or normative prescriptions grounded on 
existing literature. 

19.3% 

Exploratory 

Study 

This research makes observation of [Acquisition] for the 
purposes of developing theories, but leaves the testing of the 
theories for other studies.It means that the researcher starts 
by gathering as much information about the [subject] and 
jotting everything down until he gets a better picture of what is 
necessary (theory). There is no formal hypothesis, its purpose 
is to explore some areas more thoroughly to develop some 
specific hypothesis or prediction than be tested in future 
research. 

26.6% 

Methodology 

It is a review of research methodologies used in the field of 
[Acquisition], typically in “how-to” form. It introduces an 
academic research methodology as well as a practitioner 
methodology. 

15.9% 

Hypothesis 

Testing 

When a researcher wants to test, the researcher starts with a 
hypothesis. A hypothesis is a specific statement of predictioni^ 

9.9% 


Table 2. Type of Research Performed^o 

The most common type of research performed was Normative review. 
“Civil-Military Integration: The Context and Urgency” (Article #102, Appendix A) by 


John Mentzer and Kenneth A. Kahn, “A Framework of Logistics Research,” Journal of Business 
Logistics, Vol. 16, No. 1, 1995, pp. 240-241. 

17 Ibid. 

1^ William M. K. Trochim, “Deductive and Inductive Thinking,” http://www.socialresearchmethod. 
net/kb/dedind.htm, 20 October 2006, last accessed on 12 December 2006. 

m William M. K. Trochim, “Hypotheses,” http://www.socialresearchmethod.net/kb/hypothes.htm, 
20 October 2006, last accessed on 12 December 2006. 

29 Craig Carter and Lisa Ellram, “Thirty Five Years of the Journal of Supply Chain Management: 
Where Have We Been and Where are We Going?,” Journal of Supply Chain Management, Spring 2003, 
p. 31. 
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William B. Scott, which talked about the need for a strong industrial base to maintain our 
economic and military strength, is an example of Normative review. A thin line appears 
to exist between Normative and Literature reviews. In the case of 
Dr. William Washington, the coders classified some of his articles as Normative review, 
while others were classified as Literature review. With contracting as the main theme for 
all of his contributions. Dr Washington performed Literature reviews on the comparison 
of Competition and Sole Source Procurement processes (Articles #50, Appendix A) and 
Subcontracting (Article #107, Appendix A). His Normative contributions include articles 
on approaches to “Reward Contracting” (Article #57, Appendix A) and “Participatory 
Contracting” (Article #114, Appendix A). 

Only 1.3 percent separated Normative Literature from Exploratory Studies. 
Mr. Richard Sylvester and Mr. Joseph Ferrara’s contribution in the winter 2003 issue, 
“Conflict and Ambiguity Implementing Acquisition,” explored the struggles of 
implementation due to policy ambiguity that drove organizational conflict (Article #164, 
Appendix A). In Table 2, Hypothesis Testing was the least common research type. 
Examples include studies on cost overruns, estimate at completion (EAC) discussions, 
lean strategies, and acquisition reform initiatives. 

C. TYPES OF RESEARCH DESIGN 

In Table 3, the types of research designs identified in this study are noted, along 
with their respective frequency of usage The agreement rate for this section was 
96 percent, with the use of a third coder and discussion to settle any disagreements. 
Based on the frequency, the majority of the articles were categorized as Case Studies. 
Some authors did Case Studies on particular systems such as Joint Stars (Airborne 
Standoff Target Acquisition Recognition System),2i M102 Howitzers,^^ 
Naval Air Systems Command (NAVAIR) team and F/A-18 integrated product teams 


Article #12, “Coming Up Golden: Defense Acquisition Board Review Guide for Program Offices,” 
Appendix A. 

22 Article #30, “The Impact of Technical Data Transfers Problems during a Transition of Weapons 
System Production between Nations,” Appendix A. 
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(IPT),23 the Defense Contract Management Command (DCMC)^^ in Sealy, Texas, the 
DAU consortium,25 and Marine Corps Maintenance Center.26 Not far behind 
Case Studies were Topic Presentations. Experiments and Simulation, as well as 
Mathematical Models, were tied for the least common type of research design. In 
“Acquisition Reform Theory and Experimental Evidence for Tournament Sponsors,” the 
four authors performed an experiment suggesting that carefully designed research 
tournaments can be highly effective at promoting research efforts .22 Dr. Mark Nissen, 
the author of “Reengineering the REP Process through Knowledge-Based Systems,” 
illustrated “the use and utility of a knowledge-based systems to support process redesign 
are demonstrated, and insight is provided into the potential of [Artificial Intelligence] 
(Al)-based technologies to dramatically improve military procurement.”28 The last 
category in this section, the Mathematical Model, was represented by two authors who 
co-wrote two sequels to a specific issue on risk. In 2003, Mr. Paul Garvey and 
Mr. Chien-Ching Cho wrote an article titled “An Index to Measure a System’s 
Performance Risk” (Article 174, Appendix A) and in 2006, it was followed by “An Index 
to Measure a System-of-System’s Performance Risk” (Article 225, Appendix A). Both 
mathematically-oriented, these two articles showed computations of the technical risk 
index (TRI) to improve technical performance by lessening risk. Additionally, the 
articles were almost identical, except that the second article added focus on 
system-of-systems (SoS). 


23 Article #72, “Team-Based Redesign as Large-Scale Change: Applying Theory to the 
Implementation of Integrated Product Teams,” Appendix A. 

24 Article #77, “Concept of Operations and Implementation Plan for Industry Integrated Logistics 
System (I2LS),” Appendix A. 

25 Article #83, “A Case Study for the Systems Approach for Developing Curricula: ‘Don’t Throw Out 
the Baby with the Bath Water,’” Appendix A. 

26 Article #187, “Applying Theory of Constraints Principles and Lean Thinking at the Marine Corps 
Maintenance Center,” Appendix A. 

22 Article #95, “Acquisition Reform Theory and Experimental Evidence for Tournament Sponsors,” 
Appendix A. 

28 Article #41, “Reengineering the REP Process Through Knowledge-Based Systems,” Appendix A. 

14 



Category 

Definition 

Frequency 

Archival Data 

Data that already exist and that have been collected by 
others. This archival data is the simplest kind of data to 
gather because someone else has already done the work. 

9.9% 

Case Study 

In-depth data are gathered pertaining to a program or event. 

39.9% 

Experiment and 
Simulation 

Data collected through an experimental process or through 
a simulation process. Due to the infrequent occurrences, 
we combine them into one category. 

3.9% 

Interviews 

Data collected through conduct of interviews. 

9.9% 

Mathematical Model 

Data collected and explained using a mathematical model. 

3.9% 

Surveys 

Data collected through surveys. 

7.3% 

Topic Presentation 

No discernable design methodology. 

25.3% 


Table 3. Types of Research Design29 

These results indicate that of the 233 articles, approximately 60 percent have been 
qualitative studies and not quantitative- a figure that agrees with Elder’s study. 

D. TYPES OF DATA ANALYSIS 

Table 4 has a list of the types of data analysis that the coders used to describe 
each of the articles. The coders were in agreement 92 percent of the time. Again, a third 
coder was used to assist with any disagreements between the primary coders. 


^9 Mitchell J. Elder, Capt, USAF, “An Eleven-Year Retrospective of Acquisition Review Journal,” 
Master’s thesis, Air Force Institute of Technology, 2005, p. 12. 
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Category 

Definition 

Frequency 

Anecdotal 

Evidence 

It is an informal account of evidence in the form of an 
anecdote or hearsay. It is based on incidental observations 
on reports rather than on systematic evaluation. The 
argument draws a conclusion from cases specifically 
chosen to support the conclusion. 

10.7% 

Comparison 

Analysis 

This type compares two or more things (e.g., DoD versus 
commercial organizations). 

29.2% 

Content 

Analysis 

When we say content analysis, the researchers use detailed 
systematic evaluation of a particular body of material for 
the purpose of identifying patterns, themes, or biases.30 

35.2% 

No Analysis 

No particular analysis for the article. 

15.5% 

Statistical 

Analysis 

To analyze the data, researches use statistical models and 
techniques. This category may include correlation, 
regression, descriptive statistics, and means testing. 

9.4% 


Table 4. Types of Data Analysis^i 

Content Analysis was the most frequent type of analysis at 35.2 percent, and is 
apparent in several articles in Appendix A. Comparison and Content Analysis comprise 
approximately 65 percent of articles that were published in the ARJ. This again suggests 
that this study is still performing more qualitative, rather than quantitative, analysis. 
Statistical Analysis is the least preferred method noticed in the ARJ. Statistical Analysis 
includes various techniques such as Analysis of Variance (ANOVA), empirical analysis, 
descriptive statistics, and means testing. ANOVA was employed by the authors of 
“Investigating the Integration of Acquired Firms in High-Technology Industries” (Article 
#178, Appendix A) “to identify the significance and individual level of variance of the 
different variables based on a firm’s environment and the timing of acquisition.”^^ A 
Descriptive Statistics example is the article on the “Relationship of Cost Growth and 
Schedule Growth” (Article 169, Appendix A), where the data points from the RAND 
Cost Growth, Selected Acquisition Report (SAR) were analyzed using the mean, standard 

30 Mitchell J. Elder, Capt, USAE, “An Eleven Year Retrospective of Acquisition Review Journal,” 
Master’s thesis. Air Eorce Institute of Technology, 2005, p. 13. 

31 Ibid., p. 14. 

32 John Driessnack and David King, “Investigating the Integration of Acquired Eirms in 
High-Technology Industries,” Acquisition Review Journal, Summer 2003, p. 269. 
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deviation, coefficient of variance (CV), and the percentiles. The article concluded by 
finding no correlation between cost and schedule growth.33 in the article “The Use of 
Performance Incentives in DoD Contracting” (Article 75, Appendix A), the authors 
employed empirical analysis to establish the relationship between the performance 
achieved by contractors and certain variables such as cost sharing ratio, cost, and 
target profit. 

E. STATISTICAL METHODS 

The Ch-Squared Test provides a technique to test the significance of the 
respective categorical analysis across time periods. This is achieved by calculating 
expected values using the proportionality method in which the number of a certain 
category (i.e., type of research design) in a certain time period is assumed to be 
proportional to the amount of the total number of that same category in the total time of 
the publication. Because the Chi-Squared Tests become unstable when categories 
contain few counts, three time periods were used rather than the standard four time 
periods used throughout the rest of the analysis in order to achieve a larger sample size 
per category, thus enabling the analysis. 

F. THEMES 

In order to create a consolidated list of major themes for the ARJ, both coders 
took note of what they felt the theme(s) were for each of the 233 articles. The themes 
were extracted from the main Excel spreadsheet, assigned to a single cell, and sorted 
alphabetically to determine recurring themes. At first, the list consisted of 216 themes, 
but through further discussion was eventually reduced to a list of 27. Themes of a similar 
nature were grouped together. A list of the theme categories and subcategories are listed 
in Table 5. Each one of the 233 articles was assigned to one of the 27 theme categories. 
Eigure 1 in Chapter III shows the frequencies of the themes over the 13-year existence of 
the ARJ. 


33 Richard L. Coleman, Jessica R. Summerville, and Megan E. Dameron, “The Relationship between 
Cost Growth and Schedule Growth,” Acquisition Review Journal, Spring 2003, p. 120. 
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Themes 

Subcategories 

Technology 

Anti-tamper, Information Technology, Innovation, Knowledge 
management, Net-Centric 

Acquisition Reform 

Continuous Improvement, Restructuring, Downsizing, 

Transformation 

Contracting 

Auctions, Buyer-Seller Relationship, Contracting Officer’s 
Technical Representative (COTR) Expert System Aid (CESA), 
Contract Management, Support, Contractor Performance, Contract 
Types, Incentives, Price-Based Acquisition, Request for Proposal, 
Service-Level Agreements, Sole Source, Subcontracting, 
Competition 

Cost, Schedule, Performance 

Management, Estimation, Activity-Based Costing, Budget, Cost as 
an Independent Variable (CAIV), Cycle-time, Estimate at 
Completion (EAC), Earned Value Management (EVM), Total 
Ownership Cost (TOC), Management Reserve 

Organizational Behavior 

Cultural Change, Human Resources Management, Leadership, 
Organization Dynamics, Radical Change, Trust, Workforce 

Education and Training 

Defense Acquisition Workforce Improvement Act (DAWIA), 
Advanced Education, Certification, Community of Practice, 
Instructional System Design 

Logistics 

Electronic Business, Depots, Life-Cycle, Logistics Reform, Supply 
Chain, Performance-Based Logistics (PBL), Technical Performance 
Risk Index (TPRI) 

Risk 

Management, Risk Models, Metrics, Cost, Schedule, Performance, 
Technology 

Software 

Cyber warfare. Development, Metrics, Modularity, Standardization 

Streamlined Acquisition 

Consolidation, Restructuring 

Research, Development, Test, and 
Evaluation 

Technical Performance Measures, Test and Evaluation, Technical 
Evaluation 

Systems Engineering 

DoD Architecture Framework (DoDAF), Enterprise Architecture, 
Integration, Quality Assurance, Quality Improvement, Trade-Off 

Laws, Policies, and Regulations 

Buy America Act, Copyright, National Security 

Management 

Management Related Items 

Change 

Dynamics 

Defense Industry 

Budget, Security, Aircraft, Commercial Off the Shelf (COTS), 
Non-Developmental Items (NDIs), Small Business, Weapon 
System Acquisition 

Procurement 

Purchasing 

Decision Making 

Decision-Making Process, Quantitative Decision Analysis, 
Strategic Planning 

International 

Cooperative Acquisition, Foreign Military Sales, Globalization, 
Transatlantic 

Program Management 

Program Executive Office/r (PEO), Defense Acquisition Board 
(DAB) 

System of Systems 

Interoperability 

DoD Framework 

DoD 5000 

Lean Implementation 

Lean Six-Sigma 

Industry 

Industrial and Commercial Capability 

Modeling and Simulation 

Simulation-Based Acquisition 

Evolutionary Acquisition 

Spiral Development 

Table 5. 

Theme Categories 
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In the winter 1997 issue, after approximately four years of publication, 
Defense Acquisition was defined as: 

. . . the conceptualization, initiation, design, development, logistics 
support, modification, and disposal of weapons and other systems, 
supplies, or services to satisfy Defense Department needs, or intended for 
use in military missions.^4 

Another substantial factor that could have an overall impact on the themes started 
with Issue 38, when the ARJ began concentrating on themed issues. Some of those 
themes include Change, Risk Management, and Transformation. 

Based on the data collected, the coders were able to perform various analyses 
(which are presented in Chapter III), including the results of the comparison to Elder’s 
thesis as well as any changes that occurred between 2004 and 2006. 


^^Acquisition Review Journal, “Guidelines for Authors,” Winter 1997, p. 127. 
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III. RESEARCH ANALYSIS 


A. THEMES 

The first item of interest is the identification of themes throughout the existence 
of the ARJ. Analysis was performed to determine if trends were formed based on 
“hot issues” within the Acquisition community. Part of this analysis was to discover if 
there were any “hot button” issues discussed over time. How much has been written 
about Acquisition reform? How prominent were schedule slips and budget issues within 
the publication? Are the themes very specific or very broad across the articles published 
in the ARJl Which areas of the Acquisition community are under-represented or 
under-explored? 

The themes collected across the years were tabulated to account for their 
frequency of usage and to explore any patterns. Although several of the articles had 
multiple themes and could be assigned to more than one theme category, it was decided 
to assign each article to one theme category—that which best exemplified its main focus. 
Some similarities exist between this study’s list of themes and Elder’s list, but not 
unexpectedly, the lists were not exact replicas of each other. Twenty-seven categories 
were agreed upon by the two researchers, while Elder’s list was comprised of 15 themes. 
Doubling the number of bins available, this study has more flexibility to spread out its 
articles more specifically than Elder’s categorical themes. In Appendix E, Reform 
Initiatives was his most common theme versus this study’s category of Technology. In 
this study, the theme of Cost, Schedule, and Performance is ranked second. Referring 
back to Appendix E, when Cost and Schedule combines with Performance and 
Management, Elder’s number is very close to what this study arrived at. Eooking for 
more similarities, the number of occurrences for Elder’s Reform Initiatives theme is equal 
to the number of occurrences for the current study’s Acquisition Reform and Streamline 
Acquisition themes. Eigure 1 shows the number of articles contained within each theme 
category in descending order. As a final means of studying the themes in the ARJ, a 
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sampling of articles is provided in Appendix B to better illustrate the themes selected 
for categorization. 


Technology 
Cost, Schedule, Performance 
Contracting 
Acquisition Reform 
Education & Training 
Risk 
Software 
Logistics 

Laws, Policies, and Regulations 
Organizational Behavior 
Lean Implementation 
International 
SE 
RDT&E 
Management 
Defense Industry 
Change 

Streamline Acquisition 
Program Management 
Modeling and Simulation 
System of Systems 
DoD Eramework 
Decision Making 
Procurement 
Industry 

Evolutionary Acquisition 
IPPD/IPT 

0 5 10 15 20 25 30 

Frequency 



Figure 1. Themes Categories 

Technology is the most common theme thus far in the ART, with Cost, Schedule, 
and Performance close behind. Although Issue 38 started dedicating themes to each 
issue, neither Technology nor Cost, Schedule, and Performance were one of those issues, 
leading to the belief that themed issues have yet to impact the trends presented thus far. 
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Starting in 2005, all ARJ issues were assigned a specific theme for the authors to 
concentrate on in order to be published. If the ARJ continues to publish issues based on 
these themes, then, in the long run, this decision will have an impact on the categorization 
of themes. 

Elder broke his pool of 193 articles into three periods in order to search for trends 
over time. His analysis shows Management and Organizational Behavior and 
Interoperability increased across the time periods, while a majority of the other 
categories show either a decrease or a steady number of articles across periods. The 
difference in categories were apparent; however, the researchers believed it was based on 
the methodology used to derive their top theme categories, which were primarily based 
on interpretation and probable differences in their professional background and 
educational experience. 

Figure 2 shows the cumulative frequency of the occurrences of the 233 themes in 
the different time periods. Themes represented consistently across time periods include 
Technology, Cost, Schedule, and Performance', Contracting', Acquisition Reform', 
Education and Training', and Risk. Less than a third of the themes were localized to one 
time period. For example, although the concept of lean implementation has been 
important in recent years, authors did not start writing about it until after 2004. On the 
other hand, the Federal Acquisition Streamlining Act was enacted in 1994, and articles 
about Streamlining Acquisition are found exclusively in time period 1, which ended in 
1997. Laws, Policies, and Regulations were written most extensively during the first four 
years of the 13-year time period. The bulk of Risk occurred in time period 3 and 
Logistics in time period 4. Evolutionary Acquisition and System-of-systems emerged as 
topics during the third period, suggesting their increased importance in defense 
acquisition. Acquisition Reform decreased as a theme with each subsequent time period. 
Perhaps this is because acquisition reform is now being tailored to specific areas; 
therefore, it may not be prevalent as the main theme, but it still exists within the 
community. DoD Eramework and Decision Making were topics of choice in period 1; 
both of which are making a comeback in period 4. 
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Figure 2. Themes in Time Periods 

Figure 2 depicts a picture of the themes per individual time period. The most 
frequently-occurring theme was Technology during period 3. In period 2, Contracting 
was the most recurring theme, thus ranking second across all four time periods. Logistics 
consistently increased over the time periods, as there was one article in period 1, after 
which the number of articles doubled in period 2, and then tripled as the theme made a 
comeback in the fourth time period. 
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B. CONTRIBUTING AUTHORS 


The researchers felt it necessary to give credit to the authors who have contributed 
to the ARJ. First, the study revealed that a total of 325 authors have contributed articles 
to the ARJ. Based on the number of articles contributed by each author, 90 percent 
contributed only one article. Table 6 is a list of top contributors for 193 articles, which 
reflects three percent of the authors and matches the sample size in Elder’s thesis, thus 
allowing direct comparisons. These totals do not reflect special issue introductions and 
editorial editions. 


Author(s) 

Contributions 

Christensen, David S. 

7 

Washington, William N. 

6 

Alford, Eionel D. 

4 

Nissen, Mark E., Dr. 

4 

Graham, Robert 

3 

Einster, Bruce G. 

3 

Pollock, Neal 

3 

Templin, Carl 

3 


Table 6. Top Contributors for 193 Articles 

Compared to Elder’s thesis, there were a few minor differences which are 
believed to be due to data collection methodology. In the current study, the two 
researchers each maintained a separate, single spreadsheet. Each author and co-author 
occupied an individual cell, allowing the researchers to perform a sort function in Excel, 
which provided them with an accurate count of the authors. Since the researchers 
maintained their own spreadsheets, the chance of error was decreased. Upon completion 
of the sort function, the results were identical. The researchers suspect Elder might have 
used a manual count method, leading to the slight differences in totals. Some of the 
leading authors were co-authors in a few articles, so the chance of overlooking those 
authors is highly possible. Elder’s results can be viewed in Appendix D. Several 
observations were as follows. Dr. David Christensen was the top contributor according 
to both studies. However, notice that this study counted seven articles for this author, 
while Elder found six articles in which included an issue introduction. 
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Dr. William Washington contributed six articles according to this study; but according to 
Elder’s, he contributed five articles. On the other hand, this study showed four article 
contributions for Dr. Mark Nissen, while Elder noted five contributions. The 
contributions of Et Col Eionel Alford (four articles), Et Col Bruce Einster, 
Mr. Neal Pollock, and Dr. Carl Templin (three articles) agreed in both studies. 
Mr. Robert Graham was included among the top contributors because of his submission 
of three articles. Elder’s list of top contributors included the following authors: 
Maj Joseph Besselman, Et Col Driessnack, Dr. Keith Snider, Mr. Ashish Arora, and 
Mr. Patrick Earkey; however, they are not reflected in the current list. All authors 
mentioned were noted with three articles in Elder’s study, but only two articles were 
found for each author in this study. 

The number of authors that contributed to the ARJ did not change substantially 
from 2004 to 2006. Table 7 represents the Top Contributors for 233 articles. 
Daniel J. Sherman joined Mark E. Nissen and Eionel D. Alford in the top four authors, 
but otherwise the list was unchanged from Table 6. 


Author(s) 

Contributions 

Christensen, David S. 

7 

Washington, William N. 

6 

Alford, Eionel D. 

4 

Nissen, Mark E., Dr. 

4 

Sherman, Daniel J. 

4 

Graham, Robert 

3 

Einster, Bruce G. 

3 

Pollock, Neal 

3 

Templin, Carl 

3 


Table 7. Top Contributors for 233 Articles 


Table 8 shows the number of authors per article per time period. An overall 
picture shows that roughly 60 percent of the articles were written by a single author. 
Twenty-two percent of the articles were tendered by two authors, 15 percent had 
three authors, and the remaining had four or more authors counting one article out of 233 
that had five authors. The mean (average number) of the authors per article was 
calculated based on the number of contributing authors and the number of articles 
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published each year. For example, in 1994, 21 authors contributed to the ARJ and 
18 articles were published; therefore, there were 1.17 authors per article. Prior to 2001, 
the number of articles written by one author remained steady. In 2002, it dropped by 
more than half and there was an increase in the number of two or more authors per 
article. Elder’s results up until 2004 were relatively similar to these results. 


Year 

No. of 
Authors 

No. of 
Articles 

Mean 

One 

Author 

Two 

Authors 

Three 

Authors 

Four + 
Authors 

1994 

21 

18 

1.17 

83.3% 

16.7% 

0.0% 

0.0% 

1995 

14 

10 

1.40 

70.0% 

20.0% 

10.0% 

0.0% 

1996 

13 

11 

1.18 

81.8% 

18.2% 

0.0% 

0.0% 

1997 

44 

25 

1.76 

60.0% 

8.0% 

28.0% 

4.0% 

1998 

32 

20 

1.60 

60.0% 

25.0% 

10.0% 

5.0% 

1999 

38 

24 

1.58 

62.5% 

20.8% 

12.5% 

4.2% 

2000 

39 

22 

1.77 

63.6% 

13.6% 

9.1% 

13.6% 

2001 

18 

14 

1.29 

78.6% 

14.3% 

7.1% 

0.0% 

2002 

42 

19 

2.21 

31.6% 

26.3% 

31.6% 

10.5% 

2003 

30 

17 

1.76 

52.9% 

17.6% 

29.4% 

0.0% 

2004 

27 

13 

2.08 

23.1% 

46.2% 

30.8% 

0.0% 

2005 

50 

28 

1.79 

46.4% 

35.7% 

10.7% 

7.1% 

2006 

17 

12 

1.42 

58.3% 

41.7% 

0.0% 

0.0% 


Table 8. Authors per Article per Time Period 

As found in Carter and Ehram’s article, the “significantly greater number of 
authors per article over time is similar to the pattern that has been documented in other 
academic discipline.”^5 According to them, researchers’ thought process is the level of 
productivity is greater when the responsibility is shared by fellow researchers.^6 

An analysis of the number of contributing authors per article during the ARJ’s 
13-year existence followed using ANOVA and the Chi-Squared Test. The researchers 
maintained Elder’s breakdown of four time periods. Eor both techniques, the 
hypotheses were: 

Hq: No difference exists in the number of authors per article across time periods. 

Hi: A difference exists in the number of authors per article across time periods. 


Craig R. Carter and Lisa M. Ellram, “Thirty-Five Years of the Journal of Supply Chain 
Management: Where Have We Been and Where are We Going?,” Journal of Supply Chain Management, 
Spring 2003, pp. 37-38. 

36 Ibid., p. 38. 
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For the ANOVA technique, the dependent variable was the number of authors per 
article, while the independent variable was the time periods ranging from time period 
1 = 1994-1997, period 2 = 1998-2000, period 3 = 2001-2003, and period 4 = 2004-2006. 
Appendix H supports the null hypothesis; there is not enough evidence to say differences 
exist in the number of authors per article across time periods (p = 0.1115 and 
F = 2.0228). 

On the other hand, the Chi-Squared Test revealed a different result. For the 
expected frequencies, the researchers made an assumption that the number of 
one-authored articles in a particular time period is proportional to the number of 
one-authored articles in the total number of articles for all time periods. For example, 
64 one-authored articles out of 233 total articles yield 27.5 percent. If this assumption 
were true, then in time period 1, 27.5 percent of the 136 articles (total for time period 1) 
yields 37 expected one-author articles. With a p-value of 0.0099, the researchers have 
enough evidence to reject the null hypothesis, meaning there is a difference in the number 
of authors across time periods (Appendix K). 

Again, this study is interested in whether its findings agreed with Elder’s findings. 
Elder’s study was replicated by dividing the present data into identical time periods in 
line with his time period divisions, which consisted of a four-, three-, and four-year 
breakdown. Time periods were period 1 = 1994-1997, period 2 = 1998-2000, and period 
3 = 2001-2004. 

Referring to Appendix E, for the identical time period, the ANOVA analysis 
exposed a difference in the number of authors per article per time period (p = 0.0341). 
While Elder’s p-value was 0.0205, he also had the same findings- that the results suggest 
a statistically significant overall difference across time periods. The difference in 
p-values probably relates to a minor counting difference in the number of authors; recall 
that this study’s count of leading authors differed from Elder’s count and those 
differences might reflect a common cause of slightly different counts for numbers of 
authors. Both findings discovered a similar upward trend, as did Carter and Ellram’s in 
their study of the Journal of Supply Chain Management, where the mean of authors per 
article increased over time. 
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Furthermore, we examined the difference between the results of the ANOVA for 
the articles through part of year 2004 (Elder’s sample) and the articles until year 2006. 
Referring back to Table 8, the means were elevated: of 2.21 for 2002 and 2.08 for 2004. 
These two means joined together in the same time period (2001-2004), as replicated from 
Elder’s study) elevated the mean for the second (through 2004) ANOVA analysis. In 
part one (until 2006), where 233 articles were counted, the two means were spread out in 
two time periods of three and four. Therefore, the variability was spread out thinner 
across six years. 

Eigure 3 shows the number of articles published per year (year 2006 is not a 
complete year). It does not show a noticeable trend of increasing number of articles over 
time, although there were significant spikes from 1996 to 1997 and 2004 to 2005. There 
does not appear to be any relationship between the maturation of the ARJ and the number 
of articles being published. 



Eigure 3. Number of Articles per Year 

1. Author’s Institutional Affiliation 

ARJ solicits from any prospective authors who have published and have expertise 
in the subject and who are involved in the defense acquisition process. Table 9 shows the 
organizations included under each of the coder’s final categories. 
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Institutional Category 

Inclusion 

Civilian Universities 

All civilian academic universities and colleges, both 
graduate and post graduate levels 

Civilian Organizations 

Non-DoD and non-services organizations 

Military Universities 

Includes AFIT, NPS, USAF Academy, Naval Academy, 
Army Academy, and U.S. Military Academy 

Other USA 

Non-academic U.S. Army facilities and organizations 
i.e.. Army Research Laboratory’s Aerodynamics 

Branch, U.S. Army Corps of Engineer, and 1-36 

Infantry 

Other USAF 

Non-academic U.S. Air Force facilities and 
organizations, i.e., F/A 22 Systems Program Office 

SPO, Joint Strike Fighter JSF SPO, and Special 

Operations Command 

OSD 

Office of Under Secretary of Defense, Acquisition, 
Technology, and Logistics (USD (AT&L)), Weapons 
Systems Cost Analysis Division, and Management 

Policy and Program Integration Division 

ACSC/DSMC 

Faculty and students of Air Command Staff College and 
Defense Systems Management College 

Other DoD 

Includes Ballistic Missile Defense Organization 

DAU 

Faculty and students of DAU 

AWC/ICAF/NWC 

Faculty and students of Air War College, Industrial 
College of the Armed Forces, and Naval War College 

Other USN 

Non-academic U.S. Navy facilities, i.e.. Naval Center 
for Cost Analysis, Department of the Navy, Chief 
Information Office, and Office of Naval Operations 

Other GOV 

General Accounting Office, National Aeronautics Space 
Administration (NASA), and Atlantic Council of the 
United States 

Other USMC 

Non-academic U.S. Marine Corps facilities, i.e.. Marine 
Corps Systems Command, Quantico, VA and Depot 
Maintenance Business Area, HQ Marine Corps 

Civilian Research 

Institutions 

Includes MITRE Corporation, International Research 
Institute, and Nichols Research Lab 

Table 9. 

Institutional Categories’ Inclusions 


Figure 4 shows that most contributors come from civilian universities, followed 
by civilian organizations. Military universities such as the Air Force Institute of 
Technology (AFIT), Naval Postgraduate School (NPS), Air Force Academy, and the 
Naval Academy made up 13 percent. Contributions from civilian research institutions 
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such as MITRE Corporation, Nichols Research Lab, and International Researeh Institute 
made up less than two pereent. The United States Navy (USN) and the United States 
Marine Corps (USMC) eontributed approximately two pereent, six to ten pereent lower 
than the United States Air Foree (USAF) and United States Army (USA). 


Institutional Categories 



0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% 16.0% 18.0% 20.0% 
Percent Frequency 


Figure 4. Author’s Institutional Affiliation 

In Table 10, the researehers wanted to diseover if there were evident trends based 
on the institutional affiliation of the contributing authors. (Note: Authors who have 
revealed their affiliation.) 
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Institutional Categories 

Period 1 
1994-1997 

Period 2 
1998-2000 

Period 3 
2001-2003 

Period 4 
2004-2006 

Civilian Universities 

10 

25 

12 

17 

Other USA 

14 

7 

14 

7 

Military Universities 

18 

12 

12 

5 

Civilian Organizations 

10 

14 

3 

11 

Other USAF 

2 

16 

4 

10 

Office of Secretary of Defense (OSD) 

7 

5 

3 

7 

Air Command Staff College/Defense 
Systems Management College 
(ACSC/DSMC) 

12 

8 

2 

1 

Other DoD 

3 

2 

2 

2 

DAU 

2 

0 

0 

10 

Air War College/Industrial College of 
the Air Force/ Naval War College 
(AWC/ICAF/NWC) 

4 

0 

1 

1 

Other USN 

2 

3 

2 

0 

Other Government (GOV) 

1 

1 

1 

1 

Other USMC 

1 

2 

0 

3 

Civilian Research Institutions 

3 

1 

0 

2 


Table 10. Institutional Categories Divided into Time Periods 

The DAU had an increase in contributions during the fourth period. Military 
communities started off strong in the first three periods and then experienced a dramatic 
decrease, whereas the Civilian Universities were very dedicated contributors over all four 
periods. Catering to the higher level of the Acquisition Corps, it is ironic that DoD did 
not contribute many articles. Elder’s findings display the same results and there are no 
significant discrepancies (see Appendix G). 

2. Author’s Educational Affiliation 

Out of the total number of authors, 243 revealed their highest degree of education: 
122 authors have doctorates, 110 have master’s degrees, and 11 have bachelor’s degrees. 
In their analysis, the researchers also noted those authors with multiple degrees. 
Twenty-six authors have more than one master’s degree, which is roughly 10.7 percent of 
the authors who noted their degrees. 
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Table 11 lists the Universities and Colleges of those authors who noted their 
highest academic degrees. Industrial College of the Armed Forces (ICAF) graduates 
have the largest contribution, followed by graduates of the Air Force Institute of 
Technology (AFIT) and the Naval Postgraduate School (NPS). 


University/College/Institutions 

Contribution 

Industrial College of the Armed Forces 

14 

Air Force Institute of Technology 

12 

Naval Postgraduate School 

12 

University of California* 

11 

University of Southern California 

11 

George Washington University 

9 

Boston University 

9 

George Mason University 

8 

Defense Systems Management College 

7 

University of Alabama 

7 

Stanford University 

6 

Texas A&M University 

6 

University of Nebraska 

6 

Georgia Institute of Technology 

5 

Rennselaer Polytechnic Institute 

5 

University of Michigan 

5 

Arizona State University 

4 

Carnegie Mellon University 

4 

Massachusetts Institute of Technology 

4 

University of Maryland 

4 

University of Virginia 

4 


*Berkeley, Irvine, and Los Angeles campuses 


Table 11. Author’s Educational Affiliation 

The civilian educational institutions appear to be the front-runners for published 
articles in the ARJ, although the top three are military institutions. 

3. Academe or Practitioner 

The ARJ welcomes authors from academe and those in practice. The authors 
were classified into categories that identify them as academe or practitioners within the 
defense acquisition industry. Consistent with the other elements of this study, the time 
periods were divided into three time periods and four time periods. Table 12 ends in 
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period 3, replicating Elder’s thesis. His results are located in Appendix H. Table 13 
shows the four time periods that extend to 233 articles. 


Type 

Period 1 
1994-1997 

Period 2 
1998-1999 

Period 3 
2000-2004 

Academe 

44.8% 

45.8% 

43.0% 

Practitioner 

55.2% 

54.2% 

57.0% 


Table 12. Proportion of Academe and Practitioner Contributions (193 Articles) 

On average, 55 percent of the contributors were practitioners. The results of this 
study were in line with Elder, who obtained a finding of 58 percent. 


Type 

Period 1 
1994-1997 

Period 2 
1998-1999 

Period 3 
2000-2003 

Period 4 
2004-2006 

Academe 

44.8% 

45.8% 

45.1% 

38.1% 

Practitioner 

55.2% 

54.2% 

54.9% 

61.9% 


Table 13. Proportion of Academe and Practitioner Contributions (233 Articles) 

Table 13 shows a predominance of practitioners over academe contributors. The 
increase in the magnitude of the difference in period 4 is inconclusive because 2006 is 
not representative of a full year’s publication. 

4. Civilian or Government Contributions 

Another investigative question this study attempted to answer is the frequency of 
civilian and government contributions. The ARJ is published by a defense university; 
however, it solicits articles from anyone interested in defense acquisition. Table 14 
represents the proportion of civilian and government contributors based on 193 articles. 


Type 

Period 1 
1994-1997 

Period 2 
1998-2000 

Period 3 
2001-2004 

Civilian 

25.0% 

41.1% 

49.5% 

Government 

75.0% 

58.9% 

50.5% 


Table 14. Proportion of Civilian and Government Contributions (193 Articles) 

The proportion of civilian and government contributions aligned with Elder’s 
results found in Appendix I. Government contributed three times the amount of civilian 
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contributions in the first four years of publieation, but it appeared to diminish in the next 
time periods. Civilian eontributions started to ramp up as the years went on. Table 15 
shows that the additional time period for the overall study eontinued to exhibit deereasing 
eontributions from the government. 


Type 

Period I 
1994-1997 

Period 2 
1998-2000 

Period 3 
2001-2003 

Period 4 
2004-2006 

Civilian 

25.0% 

41.1% 

43.7% 

44.2% 

Government 

75.0% 

58.9% 

56.3% 

55.8% 


Table 15. Proportion of Civilian and Government Contributions (233 Artieles) 

C. CATEGORICAL ANALYSIS 

Categorieal Analysis consists of three parts. First, the researehers want to 
replicate Elder’s thesis. After that, an independent analysis will be performed ineluding 
the artieles that have been published from 2004 to 2006. Finally, a statistieal analysis 
will show the ehanges and trends diseovered. 

I. Comparison with Elder’s Thesis (2005) 

The researehers divided the years in aeeordanee to Elder’s time periods in order to 
replicate his study. Table 16 illustrates the results of the type of researeh performed. 
Elder’s results are found in Appendix M. 


Type 

Period I 

Period 2 

Period 3 

1994-1997 

1998-2000 

2001-2004 

Normative Review 

25.0% 

28.8% 

25.4% 

Eiterature Review 

20.3% 

27.3% 

14.3% 

Exploratory Studies 

28.1% 

24.2% 

23.8% 

Methodology Review 

20.3% 

10.6% 

19.0% 

Hypothesis Testing 

6.3% 

9.1% 

17.5% 


Table 16. Types of Researeh Performed 


a. Type of Research Performed 

Although the numbers between the two studies are not identieal due to 


differenees in the interpretation of the definitions (the edueational and professional 
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background of the researchers perhaps contributed to the disparity) and individual bias, it 
did not prevent the noticing of similarities between the two. One similarity is that the 
research published in ARJ is primarily qualitative rather than quantitative research. Both 
studies show that roughly 45 percent of the type of research performed has been either 
Normative or Literature reviews, and another 48 percent has been Exploratory Research 
or Methodology reviews (“how-to” papers). Another similarity is that Hypothesis 
Testing has been the least common type of research performed. It is believed that there 
has been less quantitative research noted in the ARJ because over time there have been 
many themes represented in the journal; therefore, it has not been possible to reach a 
point where researchers can begin quantitative analysis. In other words, there is 
insufficient concentration on a specific area long enough to get enough data to conduct 
solid quantitative analysis. 

b. Type of Research Design 

After categorizing the types of research performed, the types of research 
designs are reflected in Table 17. 


Type 

Period 1 

Period 2 

Period 3 

1994-1997 

1998-2000 

2001-2004 

Survey 

1.6% 

13.6% 

6.3% 

Case Study 

50.0% 

36.4% 

42.9% 

Interviews 

9.4% 

7.6% 

14.3% 

Archival Study 

3.1% 

12.1% 

12.7% 

Experiment 

0.0% 

4.5% 

4.8% 

Simulation 

1.6% 

1.5% 

0.0% 

Mathematical Model 

4.7% 

4.5% 

1.6% 

Topic Presentation 

29.7% 

19.7% 

17.5% 


Table 17. Types of Research Design 

The results for the type of research design aligned with the results of the 
types of research performed. Both studies reported that qualitative analysis (Case Studies 
and Topic Presentation) made up substantially more than 50 percent of the articles that 
were published. These design types are used to inform about and introduce topics that 
need further research. It is interesting that both Case Studies and Topic Presentations 
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have been on a steady dee line, offset by inereases in interviews, but also in Surveys and 
Archival Studies, both of which can be more quantitative; a further increase in the 
quantitative areas could be apparent in the next few years. Elder’s results can be viewed 
in Appendix M. 


c. Type of Data Analysis 

Type of Data Analysis is the third category of the content analysis and the 
results of the current study are shown in Table 18. 


Type 

Period 1 

Period 2 

Period 3 

1994-1997 

1998-2000 

2001-2004 

Anecdotal Evidence 

7.8% 

6.1% 

14.3% 

Statistical Analysis 

1.6% 

15.2% 

11.1% 

Content Analysis 

29.7% 

28.8% 

49.2% 

Comparison Analysis 

48.4% 

36.4% 

12.7% 

No Analysis 

12.5% 

13.6% 

12.7% 


Table 18. Types of Data Analysis 

This study’s results aligned with the results of the other two categories. 
Statistical Analysis did show an increase after 1998 (although a downturn in period 3 
followed the upturn in period 2). The qualitative areas (Anecdotal Evidence, Content 
Analysis, Comparison Analysis, and No Analysis) dominate the ARJ thus far, as shown in 
both studies. Elder’s results can be viewed in Appendix M. 

In all three categorical areas, the trends are similar in both studies, but the 
numbers are not distributed evenly and this is conceivably due to differences in methods 
and interpretations. The two studies could have conducted their data collection 
differently and their process of settling any discrepancies could affect the overall 
outcome of the data. A possible methodological difference may be the manner in which 
the articles were read and categorized. The text of each article was reviewed by the two 
researchers that resulted in two different sets of results. 

2. Aggregate Analysis of 233 Articles until 2006 

This section reflects the results of 233 articles that have been published in the 

ARJ. As the publication went on beyond Elder’s initial study, the researchers want to 
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determine if the trends remained or shifted in a different direction. In this section, due to 
the increase in the number of articles (233 articles), dividing into four time periods is 
more appropriate. 


a. Type of Research Performed 


Figure 5 shows the trends of the type of research performed based on 233 
articles. Throughout the periods Normative review has dominated. The number of 
articles involved in Normative Review combined with those representing Literature 
reviews suggests that the articles published in the ARJ have a “large degree of substantive 
justification but little subsequent theory development and testing.”^^ Although 
Hypothesis Testing has not been very popular over the periods, there was a definite 
increase in the third period,38 followed by a dramatic decrease in the fourth period. 


35.0% 


Article Category 



Period 1 
1994-1997 


Period 2 Period 3 Period 4 
1998-2000 2001-2003 2004-2006 

Time Periods 


□ Normative 

■ Literature Review 

□ Exploratory Studies 

□ Methodology Review 

■ Hypothesis Testing 


Figure 5. Type of Research Performed 


John Mentzer and Kenneth Kahn, “A Framework of Logistics Research,” Journal of Business 
Logistics, Vol. 16, No. 1, 1995, p. 241. 

Consistent with Carter and Ellram’s findings that the “use of hypothesis did indeed show a 
statistically significant increase over time.” 
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b. Type of Research Design 


In Figure 6, Case Studies are shown as the dominant research design 
across all four periods. The ARJ publishes articles that cover a wide array of topics from 
everyday issues to unique programs that survived the Acquisition Process. Archival 
Studies are the second most prominent research design in the ARJ. This emphasis on 
Case Studies especially aligns with the trend favoring qualitative data. Archival Studies 
are very useful in the Acquisition community because many practitioners learn best 
through past experiences. Since the ARJ was described as a forum to exchange opinions, 
communicate policy decisions, and maintain awareness for acquisition management 
philosophies, these trends are aligned with its mission. Consistent with both Carter and 
Ellram and Elder’s findings. Experiments, Simulations, and Mathematical Modeling were 
less utilized. 39 


Research Designs 
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□ Survey 

□ Case Study/ies 

□ Interviews 

□ Archival Study 
■ Experiment 

□ Simulation 

□ Mathematical Model 

□ Topic Presentation 


Eigure 6. Type of Research Design 


39 Craig Carter and Lisa Ellram, “Thirty Five Years of the Journal of Supply Chain Management: 
Where Have We Been and Where are We Going?,” Journal of Supply Chain Management, Spring 2003, 
p. 32. 
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c. 


Type Data Analysis 


Figure 7 shows a marked deeline in the use of Comparison Analysis, but 
Content Analysis has been eonsistent throughout the periods. Statistieal Analysis is the 
least common throughout the periods; however, it has increased since the first period. 
Although the research in the ARJ has been more qualitative that quantitative, some 
evidence supports the argument that research published in the ARJ is moving in a 
quantitative direction. 


Data Analysis 



Period 1 Period 2 Period 3 Period 4 

1994-1997 1998-2000 2001-2003 2004-2006 

Time Period 


□ Anecdotal Evidence 

□ Statistical Analysis 

□ Content Analysis 

□ Comparison Analysis 
■ No Analysis 


Figure 7. Data Analysis Method 


It is worth noting that using quantitative data does not automatically imply 
a quantitative data analysis.^o 


3. Statistical Analysis of the Categories 

Throughout this study, the 233 articles have been divided into four time periods 
for the aggregate study and divided into three time periods to do a fair comparison with 
Elder’s thesis. In order to sufficiently aggregate the quantities within categories to permit 
categorical analysis, in this part of the research, the articles were divided into three time 


Janet A. Harkness, Fons .J. R. van de Vijver, and Peter Mohler, Cross-Cultural Survey Methods, 
Wiley, Hoboken, NJ, 2003, p. 36. 
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periods. This is to achieve a better sample size (greater than five for each cell) in the 
category tables to perform a valid Chi-Squared Test. The Chi-Squared Test was 
performed to find differences, if any, in the number of respective types based on three 
separate categorical periods. The first time period consists of five years (1994-1998), the 
second time period consists of four years (1999-2002), and the third time period also 
consists of four years (2003-2006). 

For all categories, this study assumes that there are no differences in the types of 
articles, types of research categories, and types of data analysis employed across time 
periods. The respective hypotheses are stated in each category. 

a. Type of Research Performed 

Tables 19 and 20 give the actual and expected frequencies for the type of 
research performed. The values in these tables were used to calculate the 
Chi-Squared Test. 

Ho: No difference in the type of article category across time periods. 

Hp A difference in the type of article category exists across time periods. 


Type 

Period 1 

Period 2 

Period 3 

1994-1998 

1999-2002 

2003-2006 

Normative 

22 

21 

23 

Literature Review 

19 

15 

11 

Exploratory Studies 

23 

21 

18 

Methodology Review 

15 

9 

13 

Hypothesis Testing 

5 

22 

5 


Table 19. Actual Frequency for Type of Research Performed 


Type 

Period 1 

Period 2 

Period 3 

1994-1998 

1999-2002 

2003-2006 

Normative 

23 

24 

19 

Literature Review 

16 

16 

13 

Exploratory Studies 

22 

23 

18 

Methodology Review 

13 

13 

11 

Hypothesis Testing 

11 

12 

9 


Table 20. Expected Frequency for Type of Research Performed 
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Employing the same assumption, the expeeted frequeneies eonsistently 
use a proportionality method. When the null hypothesis is true, the observed results and 
the expected results should be similar. On the other hand, if the null hypothesis is not 
true, some of the observed and expected results will be different. The Chi-Squared Test 
resulted in a p-value of 0.0126. Therefore, the researchers can reject the null hypothesis; 
there is a difference in the type of article categories across time periods. 

b. Type of Research Design 

Tables 21 and 22 give the actual and expected frequency for the type of 
research design. The values in these tables were used to calculate the Chi-Squared Test. 

Ho: No difference in the type of research designs across time periods. 

Hp A difference in the type of research designs exists across time periods. 


Type 

Period 1 
1994-1998 

Period 2 
1999-2002 

Period 3 
2003-2006 

Survey and Interviews 

12 

17 

11 

Case Study/ies 

40 

28 

25 

Archival Study, Experiment, 
Simulation, and Mathematical Model 

9 

17 

15 

Topic Presentation 

23 

17 

19 


Table 21. Actual Frequency for Type of Research Design 


Type 

Period 1 
1994-1998 

Period 2 
1999-2002 

Period 3 
2003-2006 

Survey and Interviews 

14 

14 

12 

Case Study/ies 

34 

32 

28 

Archival Study, Experiment, 
Simulation, and Mathematical Model 

15 

14 

12 

Topic Presentation 

21 

20 

18 


Table 22. Expected Frequency for Type of Research Design 


The p-value for this Chi-Squared Test is 0.2739, which is greater than 
0.05. The researchers combined Archival Study, Experiment, Simulation, and 
Mathematical Modeling because the four research designs share a quantitative nature. 
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Due to the similarities, they can be combined to produce a more valid test result. 
Regardless, the Chi-Squared Test concludes that there is no difference in the type of 
research categories across time periods. 


c. Type of Data Analysis 

Tables 23 and 24 give the actual and expected frequency for the type of 
data analysis. The values in these tables were used to calculate the Chi-Squared Test. 

Hq: No difference in the type of data analysis across time periods. 

Hi: A difference in the type of data analysis exists across time periods. 


Type 

Period 1 

Period 2 

Period 3 

1994-1998 

1999-2002 

2003-2006 

Anecdotal Evidence 

7 

6 

12 

Statistical Analysis 

3 

11 

8 

Content Analysis 

23 

32 

27 

Comparison Analysis 

38 

22 

8 

No Analysis 

13 

8 

15 


Table 23. Actual Frequency for Type of Data Analysis 


Type 

Period 1 

Period 2 

Period 3 

1994-1998 

1999-2002 

2003-2006 

Anecdotal Evidence 

9 

8 

8 

Statistical Analysis 

8 

7 

7 

Content Analysis 

30 

28 

25 

Comparison Analysis 

25 

23 

20 

No Analysis 

13 

12 

11 


Table 24. Expected Frequency for Type of Data Analysis 


The Chi-Squared Test showed a p-value of 0.0003. In period 1, 
Statistical Analysis was used in only three studies.Although some statisticians prefer 
more than five frequencies in each cell, the required condition specifically applies to the 
expected frequencies. The test performed in this section still fulfilled the required 


The actual sampling distribution of the test statistic is discrete but it can be approximated by the 
chi-squared distribution provided that the sample size is large. The sample size must be large enough so 
that the expected value for each cell must be five or more. Gerald Keller, Statistics for Management and 
Economics, 7th Edition, Thomson/Brooks/Cole, 2005, pp. 554-555. 
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condition, despite the low actual count of Statistical Analysis in Period 1. Therefore, a 
conclusion can be drawn from the analysis that, based on this p-value, there is insufficient 
evidence to reject the null hypothesis. 

This chapter provided an in-depth analysis of the data, to include the 
results of the replication of Elder’s study. Chapter IV presents the conclusions of this 
study and provides recommendations for future studies. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

The ARJ is described as a platform where 

.... Acquisition Corps members and other readers from government, 
Congress, industry and academe are encouraged to use the Acquisition 
Review Quarterly (ARQ), now ARJ, as their professional forum for 
discussion and exchange of policies, research, information, and opinion.42 

The research conducted in both studies suggests that the ARJ is accomplishing 
this mission. Acquisition Reform Initiatives date back to 1961 and are still occurring to 
date; this continuing in reform is reflected in the ARJ. 

The ARJ has shown some distinctive trends based on the analysis described in 
Chapters II and III. The ARJ is still a new journal; therefore, the trends are a good 
representation of a growing community. The ARJ reflects a number of qualitative 
studies, but shows progress in the number of published quantitative studies. The trends in 
the types of research performed, research design, and data analysis are currently in line 
with the recommendations of the research community. So as long as the ARJ continues 
to publish articles of its current caliber, the acquisition community should only expect to 
see more quantitative research in future issues. 

The research shows that both practitioners and academe are publishing articles in 
the ARJ. Practitioners’ contributions are slightly greater in number than those of 
academe, but the difference is less than five percent. The results are comparable to the 
mission statement, which welcomes “Acquisition Corps members and other readers from 
government. Congress, industry and academe,”^^ which leads to the conclusion that the 
ARJ is capturing its intended audience. A majority of the articles thus far have been 
contributed by civilian universities and military organizations. Organizations such as the 
General Accounting Office (GAO), the military universities, and Canadian forces 

^2 Acquisition Review Quarterly, Editorial Mission, Summer 1994, no page number noted, 
editorial submission. 

^2 Acquisition Review Quarterly, Editorial Mission, Summer 1994, no page number noted, 
editorial submission. 
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(international) were more appropriately classified under the military organizations 
umbrella. However, the military universities are trailing civilian organizations by less 
than one percent. Again, this is a reflection of the range of the ARfs audience and the 
fact that acquisition is of interest to not only military personnel, but to the 
Acquisition Corps. 

Since 1997, the number of authors per articles has increased. It has moved away 
from just one and two authors per article and an influx of three and four authors per 
articles is apparent. This shows that collaboration is encouraged among authors. 

Overall, the ARJ is meeting its published mission statement and in the upcoming 
issues we should see an increase in articles that discuss quality, quality assurance (QA), 
and configuration management because they have been topics of interest to the 
Acquisition Corps. 

B. LIMITATIONS OF THIS STUDY 

Two main points that limited the reliability of this type of study are the human 
factors and the methodology. Human factors include the level of experience and interest 
of the researchers and the limited time frame of the study. Having done this for the first 
time, the quality of the coded judgments—despite pleasantly high inter-coder 
agreement—was not entirely dependable. The methodology from the literature reviews 
(article reading) to the coding process (categorical assignments) was subject to 
interpretations as well. 

C. RECOMENDATIONS FOR FUTURE RESEARCH 

This study should be considered a stepping stone for future research on the ARJ. 
“An explicit description of the research process is needed in order to increase the 
reliability of the research in question so as to render it possible for other researchers to 
replicate the research and its findings.Recommendations include: 


^ Gyongyi Kovacs and Karen .M. Spens, “A Content Analysis of Research Approaches in Logistics 
Research,” International Journal of Physical Distribution and Logistics Management, Vol. 36, No. 5, 2006, 
p. 385. 
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1. Use a Group of Coders to Replicate the Study 

Two important threats to the validity of qualitative conclusions are the selection 
of data that fit the researcher’s existing theory or preconceptions and the selection of data 
that “stands out” to the researcher.45 The group should represent a good sample size of 
the Acquisition Corps. 

2. In-Depth Study of the Themes 

Having read Elder’s perception on the article submission, the researchers in some 
ways, agreed with him. Elder said, “. . . the type of article submitted for consideration for 
publication was driven primarily by potential authors aware of the joumal.”46 In the 
fourth time period, the ARJ posted specific themes intended for the next issue’s 
publication. This may change the course of article submission. If this requirement is 
maintained, article submission will be filtered in every issue. 

Eooking back to the definition of defense acquisition as “the conceptualization, 
initiation, design, development, test, contracting, production, deployment, logistics 
support, modifications, and disposal of weapons systems and other systems, supplies, or 
services . . .”47 it can be safely assumed that these were adequately represented. As this 
study delved into the subject matter areas, it seemed that minimal research was devoted 
to quality and QA. 

Eor the purpose of this study, one theme was assigned per article; however, some 
articles had more than one theme. A study of the themes will provide a more 
comprehensive list of the themes discussed in the ARJ. 


45 Michael Huberman and Matthew B. Miles, Qualitative Data Analysis, 2nd Edition, 
Thousand Oaks, CA, 1994, p. 253. 

46 Mitchell Elder, Capt, USAE, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Eorce Institute of Technology, March 2005, p. 33. 

47 Acquisition Review Quarterly, “Guidelines for Authors,” Winter 2002, p. 85. 
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3. Detailed Study of the Three Categories 

Each article was assigned to one category for type of research performed, type of 
research design, and type of data analysis, when, in reality, each article could have more 
than one category for each of the three areas. In order to better understand the trends it is 
recommended that a study be done assigning more than one category to each article when 
applicable. Although more extensive, this will provide a clearer picture of the direction 
of not only the ARJ, but also the Acquisition Community. 

Although 57 percent of Literature and Normative reviews about the concepts of 
acquisition were already submitted, the ARJ still needs to continue this trend, since this is 
a growing publication and there is always a benefit derived from the knowledge shared 
by these types of research. 

The potential for different interpretations regarding the categories exists. In 2005, 
Karen Spens and Gyongyi Kovacs recommended some measures addressing the validity 
and reliability of Content Analysis. The following are taken from Table I of their 
Content Analysis written in 2006: 

• Develop clear categorization schemes and decision rules for categorization 

• Follow theoretical framework and pre-define categories for each variable 

• Ensure mutual exclusiveness, independence, and exhaustiveness 
of categories 

• Fine tune of category development during the coding process 

• Ensure reliability of coding instrument 

• Ensure the reliability of the coded data set through the use of multiple coders 

• Assess measurement reliability 

• Assess coding consistency and stability's 

Gyongyi Kovacs and Karen M. Spens, “A Content Analysis of Research Approaches in Logistics 
Research,” International Journal of Physical Distribution and Logistics Management, Vol. 36, No. 5, 2006, 
pp. 380-381. 
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Reflecting back on the data collection and coding process of this study, the two 
researchers made every effort to clearly establish unanimous interpretations of the 
categories. The regular discussions definitely helped to clear up confusion; however, 
there was always the potential for one researcher to influence the other, so consistency 
was not established. Since each article is not necessarily exclusive to only one set of 
categories, it was very difficult to come to an agreement. Compromise was necessary to 
assign categories to the articles. 

4. Genealogy of Authors 

This study provided different information about the contributing authors: their 
educational background and professional background at least. It showed that the majority 
of them came from civilian universities. In the future, research should be undertaken to 
study the valuable contributors in the acquisition field who are instrumental to the 
success of the AR7.49 Such a study will lead to an analysis of the productivity of 
contributing authors in the ARJ mainly to show that the field is growing like other 
related journals. 

D. SUMMARY 

This study has proven that ARJ has met its editorial mission and has published 
articles “in-sync” with the Acquisition Corps. Theme issues are the future of ARJ which 
could change the current trends presented in this study. If that should occur then the ARJ 
should revert back to casual models to increase quantitative studies. 


^9 Jack R. Meredith and Kwasi Amoako-Gyampah, “The Genealogy of Operations Management,” 
Journal of Operations Management, Vol. 9, No. 2, pp. 146-167. 
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APPENDIX A. ARTICLE #, YEAR, AUTHOR(S), TITLE, AND 
THEMES OE ARTICLES INCLUDED IN THE RESEARCH 


ART 

# 

YEAR AUTHOR{S) 

TITLE 

THEMES 

1 

1994 Preston, Colleen 

Acquisition Refoiin: Making it a Reality 

Acquisition Reform 

2 

1994 LaBerge, Walter B. 

Restructuring DoD: Study the High-Tech Commercial World 

Acquisition Reform 

3 

1994 Christensen, David S. 

Cost OveiTun Optimism: Fact or Fiction 

Cost, Schedule, 
Performance 

4 

1994 Allen, Gail; Yoos 11, 
Charles 

Through a Glass Darkly: The Anomaly of Streamlined 

Management 

Streamline Acquisition 

5 

1994 Morrison Jr., Robert 

Mobilizing the Defense Contracting Process 

Laws, Policies, 
Regulations 

6 

1994 Chambers, George 

Variance Analysis Within C/SCSC Programs 

Cost, Schedule, 
Performance 

7 

1994 Lynn, Larry 

The Role of Demonstration Approaches in Acquisition Reform 

Acquisition Reform 

8 

1994 Horton, Peter 

Converting the Military-Industrial Complex: Why is it difficult 

Defense Industry 

9 

1994 Templin, Carl 

Defense Contracting Buyer-Seller Relationships: Theoretical 
Approaches 

Defense Industry 

10 

1994 Shields, John T 

Factors Affecting New Product Development 

Procurement 

11 

1994 Pariseau, Richard; 
Oswalt, Ivar 

Using Data Types and Scales for Analysis and Decision Making 

Decision Making 

12 

1994 Greenburg, Harvey R.; 
Palley, Lynn B. 

Coming Up Golden: Defense Acquisition Board Review Guide for 
Program Offices 

Program Management 

13 

1994 Goodman, Sherri 

The Lemon Juice Solution: Pollution Prevention and Acquisition 
Reform 

Acquisition Reform 

14 

1994 Neilson, Robert 

The Role of Information Technology in National Security Policy 

Laws, Policies, 
Regulations 

15 

1994 Shebalin, Paul V. 

Software Development Standards and the DoD Program Manager 

Software 

16 

1994 Sutton, Jeanne C. 

Marrying Commercial and Military Technologies: A New Strategy 
for Maintaining Technological Superiority 

Technology 

17 

1994 Williams, Richard M. 

Small Business Manufacturing: An Important Component of the 

U.S. Defense Industrial Base 

Technology 

18 

1994 Glenn, Mark W. 

Using Explicit Demand Curves in an Acquisition Strategy 

Contracting 

19 

1995 Byms Jr, Edward V.; 
Corban, J. Eric; Ingalls, 
Stephen A. 

A Novel Cost-Benefit Analysis for Evaluation of Complex Military 
Systems 

Logistics 

20 

1995 Wells, Rita Lappin 

Contracting Readiness: Timely Support for Military Operations 

Contracting 

21 

1995 Scott, William, et al. 

Project Kaizen Looks at Congressional Oversight of Defense 
Acquisition Programs 

Acquisition Reform 

22 

1995 Rhoads, Dusty 

Is DAWIA worth it? An Approach to Analyzing the Impacts 

Education and Training 

23 

1995 Haimes, Yacov; 
Chittister, Clyde 

An Acquisition Process for the Management of Non-Technical 

Risks Associated with Softwai'e Development 

Software 

24 

1995 Christensen, David; 
Ferens, Daniel 

Using Famed Value for Performance Measurement on Software 
Development Projects 

Software 

25 

1995 Cancian, Mark 

Acquisition Reform: It’s not as easy as it seems 

Acquisition Reform 

26 

1995 Battershell, A. Lee 

Technology Approach: DoD Versus Boeing (A Comparative Study) 

Technology 

27 

1995 Kwatnoski, Richard 

Cooperative Acquisition Projects in the Pacific Rim 

International 

28 

1995 Dakin, Timothy 

What Every Government Employee Should Know About 

Post-Federal Employment Restrictions 

Laws, Policies, and 
Regulations 

29 

1996 Doton, Larry 

Integrating Technology to Reduce Fratricide 

Technology 

30 

1996 Schaller, Michael E. 

The Impact of Technical Data Transfers Problems during a 

Transition of Weapons System Production between Nations 

Technology 

31 

1996 Cothran, Julie 

Battle Labs: Tools and Scope 

Technology 

32 

1996 Wilson Jr., John R. 

Battle Labs: What Are They, Where Are They Going? 

Technology 

33 

1996 Hewitt, Clyde 

Getting to the On-Ramp of the Information Highway 

Technology 

34 

1996 LaBerge, Walter B. 

Cycle Time: A Military Imperative As Well 

Cost, Schedule, 
Performance 

35 

1996 Luman, Ronald R.; 

Scotti, Richard S. 

The System Architect Role in Acquisition Program Integrated 

Product Teams 

Acquisition Reform 

36 

1996 Snider, Keith F. 

DAWIA and the Price of Professionalism 

Education 

37 

1996 FeiTara, Joe 

DoD's 5000 Documents: Evolution and Change in Defense 
Acquisition Policy 

DoD Framework 

38 

1996 D'Agostino, Davi M. 

Transatlantic Cooperative Weapons Development: How can we 
Better Ensure Success? 

Integrated Product and 
Process Development/ 
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Integrated Product 
Team (IPPD/IPT) 


39 

1996 

Bregard, Richard W.; 
Chasteen, Taylor 

Implementing Integrated Product Development: A Project 

Manager’s Perspective 

International 

40 

1997 

Ciccotello, Conrad S.; 
Green, Steve G.; 

Hornyak, Martin 

Rethinking Twenty-First Century Acquisition: Emerging Trends for 
Efficiency Ends 

Laws, Policies, 
Regulations 

41 

1997 

Nissen, Mark 

Reengineering the RFP Process Through Knowledge-Based 

Systems 

Contracting 

42 

1997 

Ranquet, Robert 

Think Tanks and the National Security Strategy Formulation 

Process: A Comparison of Current American and French Patterns 

Laws, Policies, 
Regulations 

43 

1997 

Haimes, Yacov; 

Chittister, Clyde; 

Schooff, Richard 

A Holistic Management Framework for Software Acquisition 

Software 

44 

1997 

Steves, Michael R. 

Addressing Risk Management in Non-Development Items 

Acquisition Programs 

Risk 

45 

1997 

Pinker, Aron, Smith; 
Charles G.; Booher, Jack 
W. 

Chew, James S. B. 

Selecting Effective Acquisition Process Metrics 

Streamline Acquisition 

46 

1997 

Commercial Best Practices and the DoD Acquisition Process 

Streamline Acquisition 

47 

1997 

O'Connor, Michael F.; 
Paris, Janine L.; 

Lovelace, Joan S. 

A Decision Support Procedure for Best Value Source Selections 

Contracting 

48 

1997 

Peeler Jr., David L. 

Applying Dialectic to Acquisition Strategy 

Streamline Acquisition 

49 

1997 

Rush, Benjamin C. 

Cost as an Independent Variable: Concepts and Risks 

Cost, Schedule, 
Performance 

50 

1997 

Washington, William N. 

A Review of the Literature: Competition Versus Sole Source 
Procurement 

Contracting 

51 

1997 

Anderson, Timothy P.; 
Cherwonik, Jeffrey S. 

Cost Estimating Risk and Cost Estimating Uncertainty Guidelines 

Cost, Schedule, 
Performance 

52 

1997 

Barrick, Alan E.; Alberts, 
Henry C. 

Acquisition of State-of-the-Art Logistics Combat Support System: 
The Joint Logistics Advanced Concept Technology Demonstration 
Program 

Acquisition Reform 

53 

1997 

Clark, Rolf 

The Hidden Implications of Force Changes 

Laws, Policies, 
Regulations 

54 

1997 

Frank, Deborah 

A Theoretical Consideration of Acquisition Reform 

Acquisition Reform 

55 

1997 

Garcia, Andrea; Keyner, 
Hugo; Robillard, 

Thomas; VanMullekom, 
Mary 

The Defense Acquisition Workforce Improvement Act: Five Years 
Later 

Education and Training 

56 

1997 

Harman, Beryl 

The Government Performance Act: Strategic Planning of the Future 

Laws, Policies, 
Regulations 

57 

1997 

Washington, William N. 

Some New Approaches to “Reward Contracting” 

Contracting 

58 

1997 

Wollover, David R. 

Quality Function Deployment as a Tool for Implementing Cost as 
an Independent Variable 

Cost, Schedule, 
Performance 

59 

1997 

Kankey, Roland D.; 
Muczyk, Jan P.; Ely, Neal 
M 

Focused Graduation Education: An Invisible but Real Competitive 
Edge 

Education and Training 

60 

1997 

Brower, Michael J. 

Outland: The Vogue of DoD Outsourcing and Privatization 

Contracting 

61 

1997 

Hall, Mary-Jo 

Changing the Way we Assess Leadership 

Organizational 

Behavior 

62 

1997 

Davis, Randy; Phillips, 
Bill; Vazquez, Bud 

The Phoenix Rises 

IPPD/IPT 

64 

1997 

Bratt, Gary M.; 

Doganiero, Donna M.; 
Spencer, Clark 0. 

Estimating the Health Hazard Costs of Army Materiel: A Method 
for Helping Program Managers Make Informed Health Risks 
Decisions 

Cost, Schedule, 
Performance 

65 

1998 

Kock, Ned 

Government Transformation and Structural Rigidity: Redesigning a 
Service Acquisition Process 

Change 

66 

1998 

Crecine, John P.; 
Salomone, Michael D. 

A Multimedia Systems Approach to National Security Policy, 
Decision Making, and Intelligence Support 

Technology 

67 

1998 

Delano, Kenneth J. 

Identifying Factors that Contribute to Program Success 

Program Management 

68 

1998 

Ignizio, James P. 

Integrating Cost, Effectiveness, and Stability 

Cost, Schedule, 
Performance 

69 

1998 

Dobbins, James H.; 
Donnelly, Richard G. 

Summary Research Report on Critical Success Factors in Federal 
Government Program Management 

Program Management 

70 

1998 

Roberts, Nancy C. 

Radical Change by Entrepreneurial Design 

Change 

71 

1998 

Reardon, Kathleen K.; 
Reardon, Kevin J.; Rowe, 

Leadership Styles for the Five Stages of Radical Change 

Organizational 

Behavior 
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Alan J. 



72 

1998 

Hocevar, Susan Page; 

Team-Based Redesign as Large-Scale Change: Applying Theory to 

Change 



Owen, Walter E. 

the Implementation of Integrated Product Teams 


73 

1998 

Judith Gebauer; Carrie 
Beam; Arie Segev 

Impact of the Internet on Procurement 

Technology 

74 

1998 

Scacchi, Walt; Boehm, 

Virtual Systems Acquisition: Approach and Transitions 

Modeling and 



Barry 


Simulation 

75 

1998 

Hildebrandt, Gregory G. 

The Use of Performance Incentives in DoD Contracting 

Contracting 

76 

1998 

Holland, Lauren 

The Weapons Acquisition Process: The Impediments to Radical 

Organizational 




Reform 

Behavior 

77 

1998 

Boyd, James A. 

Concept of Operations and Implementation Plan for Industry 
Integrated Logistics System (I2LS) 

Logistics 

78 

1998 

Ferreiro, Larrie D. 

Organizational Trust in Naval Ship Design Bureaus: France, Great 

Organizational 




Britain, and the United States 

Behavior 

79 

1998 

McNally, William P. 

Will Commercial Specifications Meet our Future Air Power Needs? 

Acquisition Reform 

80 

1998 

Muczyk, Jan P. 

Generating Needed Modernization Funds: Streamlining the 

Cost, Schedule, 




Bureaucracy - Not Outsourcing and Privatizing - is the Best 

Solution 

Performance 

81 

1998 

Wong, Carolyn, 

An Approach for Efficiently Managing DoD Research and 

Research, 



Steinberg, Paul, Horn, 

Development Portfolios 

Development, Testing, 



Kenneth; Axelband, 


and Evaluation 



Elliot 


(RDT&E) 

82 

1998 

Childress, Alan; Larson, 

A Case for International Cooperative Acquisitions: Lessons from 

International 



James 

Developing and Executing a Section 27 “Quayle” Authority 

Program 


83 

1998 

Scafati, Anthony A. 

A Case Study for the Systems Approach for Developing Curricula: 
“Don’t Throw out the Baby with the Bath Water’” 

Education and Training 

84 

1998 

Christensen, David S. 

The Cost and Benefits of the Earned Value Management Process 

Cost, Schedule, 
Performance 

85 

1999 

Pollock, Neal 

International Cooperative Research and Development Programs 

International 

86 

1999 

Christensen, David S.; 

The Impact Of The Packard Commission’s Recommendations on 

Cost, Schedule, 



Searle, David A.; 

Vickery, Caisse 

Reducing Cost Overruns on Defense Acquisition Contracts 

Performance 

87 

1999 

Smyth, Joseph, Lt Col 

The Impact of the Buy American Act on Program Managers 

Laws, Policies, 
Regulations 

88 

1999 

Parker, Tony 

Logistics Test And Evaluation: An Overview 

Logistics 

89 

1999 

Christensen, David S. 

Value Cost Management Report to Evaluate the Contractor’s 

Cost, Schedule, 




Estimate at Completion 

Performance 

90 

1999 

Alford, Lionel D., Lt Col 

The Problem with Aviation COTS 

Defense Industry 

91 

1999 

Washington, William N. 

Depot Utilization and Commercialization 

Industry 

92 

1999 

Ainsley, Robert J.; 

DoD and the Change Paradigm: Change Agents Versus Established 

Change 



Riordan, James 

Service Roles, Missions and Cultures 


93 

1999 

Raymond, Fred 

Quantify Risk To Manage Cost and Schedule 

Risk 

94 

1999 

Alford, Lionel D., Knarr, 
Robert C. 

General Flight Test Theory Applied to Aircraft Modifications 

RDT&E 

95 

1999 

Fullerton, Richard L.; 

Acquisition Reform Theory and Experimental Evidence For 

Contracting 



Linster, Bruce G.; 

McKee, Michael; Slate, 
Stephen 

Tournament Sponsors 


96 

1999 

Traceski, Frank T. 

Assessing Industrial Capabilities For Carbon Fiber Production 

Industry 

97 

1999 

Washington, William N. 

Outsourcing Automatic Data Processing Requirements and Support 

Contracting 

98 

1999 

Besselman, Joe; Arora, 

Purchasing Performance: A Public Versus Private Sector 

Procurement 



Ashish; Larkey, Patrick 

Comparison of Commodity Buying 


99 

1999 

Huber, Arthur F; Scott, 

The Role and Nature of Anti-Tamper Techniques in U.S. Defense 

Technology 



Jennifer M 

Acquisition 


100 

1999 

Will, Edward 

Paving the Way for Price-Based Acquisition 

Contracting 

101 

1999 

Nissen, Mark E. 

SPs and Beyond: Innovating Acquisition Through Intelligent 
Electronic Contracting 

Contracting 

102 

1999 

Linscott, William B. 

Civil-Military Integration: The Context and Urgency 

Acquisition Reform 

103 

1999 

Bushey, Douglas B.; 

A Systematic Approach to Prioritizing Weapon Requirements and 

Acquisition Reform 



Nissen, Mark E. 

Military Operations Through Requisite Variety 


104 

1999 

Pinney, Charles W. 

The USAF PEO/DAC/MAD Structure Successful Pattern for Future 
Weapon System Acquisition? 

Program Management 

105 

1999 

Hanratty, Michael J.; 

Open Systems and the Systems Engineering Process 

Systems Engineering 



Lightsey, Robert H.; 
Larson, Arvid 


(SE) 

106 

1999 

Myers, Margaret 

An Investment-Based Approach for Managing Software-Intensive 

Software 


System 
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107 1999 Washington, William N. “Subcontracting” as a Solution, Not a Problem, in Outsourcing 

108 1999 Graham, Robert; Reengineering the Acquisition Process a Quantitative Example of 

Hoffman, Eric, Acquisition Reform Working for the Air Force’s Launch Programs 

System Program Office 

109 2000 Strayer, Kenneth; Hoivik, The Use of Advanced Warfighting Experiments to Support 

Thomas; Hocevar, Susan Acquisition Decisions 
Page 

110 2000 Besselman, Joseph; Measuring the Readiness Cost of One-Size-Shoe-Fits-All Public 

Arora, Ashish; Larkey, Policy: A Fact-Based Look at Cost-, Hybrid-, and Price-Based 
Patrick Purchasing 

111 2000 Brown, C. David; Grant, Building a Business Case for Modeling and Simulation 

Gordon; Kotchman, 

Donald; Reyenga, Robert, 

Szanto, Terence 

112 2000 McLlvaine, Paul J. The Evolution of 21 St Century Acquisition and Logistics Reform 

113 2000 Kaye, Michael A., Cost As an Independent Variable: Principles and Implementation 

Sobata, Mark S.; Graham, 

David R.; Gotwald, Allen 
L. 

114 2000 Washington, William N. Participatory Contracting 

115 2000 Christensen, David; An Analysis of Management Reserve Budget on Defense 

Templin, Carl Acquisition Contracts 

116 2000 Eskew, Henry L. Aircraft Cost Growth and Development Program Length: Some 

Augustinian Propositions Revisited 

117 2000 Struth, Robert G. Jr. Systems Engineering and the Joint Strike Fighter: 

The Flagship Program for Acquisition Reform 

118 2000 Truckenbrodt, Yolanda The Relationship Between Leader-Member Exchange and 

B. Commitment and Organizational Citizenship Behavior 

119 2000 Lloyd, Robert E. Government Contracting Pathologies 

120 2000 Wong, Carolyn; Maintaining the Government's Ability to Buy Smart 

Steinberg, Paul; Horn, 

Kenneth; Axelband, 

Elliot 

121 2000 Lightsey, Robert H. Engineering Management Training: Comparing Experiential Versus 

Lecture Methods of Instruction 

122 2000 Proctor, Michael; Technical Performance Measures and Distributed-Simulation 

Lipinski, Michael J. Training Systems 

123 2000 Jordon, Leland G. Systemic Fiscal Optimism in Defense Planning 

124 2000 Love, James D. Test and Evaluation Management Reform: Issues and Options 

125 2000 Pollock, Neal Management: Towards a Unified Field Theory 

126 2000 Alford, Lionel D. Cyber Warfare: Protecting Military Systems 

127 2000 Brown, C. David Enterprise Architecture for DoD Acquisition 

128 2000 Liebowitz, Jay CESA: The COTR Expert System Aid 

129 2000 Marshall, Michael L.; Private Sector Downsizing: Implications for DoD 

Hazell, Eric J. 

130 2000 Olson, Craig From Cradle to Save: Revolutionary Acquisition Force Structure 

Alternatives for the 21st Century 

131 2001 Gill, James H. Crisis in the Acquisition Workforce: Some Simple Solutions 

132 2001 Kock, Ned Changing the Focus of Business Process Redesign from Activity 

Flows to Information Flows: A Defense Acquisition Application 

133 2001 Ladymon, Joseph M. Network-Centric Warfare and its Function in the Realm of 

Interoperability 

134 2001 Zittel, Randy C. The Reality of Simulation-Based Acquisition - and an Example of 

U.S. Military Implementation 

135 2001 Angelis, Diana 1. Implementing Activity-Based Management in an Acquisition 

Organization 

136 2001 Watkins, Patrick N. The Persistence of Learning and Acquisition Strategies 

137 2001 Barzelay, Michael; How the Acquisition Workforce Adds Value 

Thompson, Fred 

138 2001 Graves, Ralph H. Seeking Defense Efficiency 

139 2001 Ciccotello, Conrad S. Complexity: A Cognitive Barrier to Defense Systems Acquisition 

Management 

140 2001 Deutch, John Consolidation of the U.S. Defense Industrial Base 

141 2001 Fowler, David N.; Innovating the Federal Acquisition Process 

Nissen, Mark E. Through Intelligent Agents 

142 2001 Nichols, Camille; The Role of Foreign Comparative Testing Programs in Army 
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Grogan, Dan; Schmidt, Modernization 
Raef 

143 2001 Bachman, David C. Single Point Adjustments: A New Definition with Examples 

144 2001 Baker, Bud Clipped Wings: The Death of Jack Northrop’s Flying Wing 

Bombers 

145 2002 Anderson, Warren M.; And the Survey says...The Effectiveness of DoD Outsourcing and 

McGuiness, John; Spicer, Privatization Efforts 
John 

146 2002 Christensen, David; EAC Evaluations Methods: Do They Still Work? 

Templin, Carl 

147 2002 Greiner, Michael A.; An Assessment of Air Force Development Portfolio Management 

Dooley, Kevin J; Shunk, Practices 
Dan L; McNutt, Ross 

148 2002 Linster, Bruce G; Slate, Consolidation of the U.S. Defense Industrial Base: Impact on 

Stephen; Waller, Robert Research Expenditures 
L. 

149 2002 Sullivan, Shannon Globalized Security: An Allied Industrial Base for the 21st Century 

150 2002 Brantley, Mark W.; Expanding the Trade Space: An Analysis of Requirements 

McFadden, Willie J.; Tradeoffs affecting System Design 
Davis, Mark J. 

151 2002 Hamilton, John A., An Interoperability Road Map for C4ISR Legacy Systems 

Summers, Paul A., 

Rosen, J. David 

152 2002 Manz, Paul C., Zelenka; Protecting Government Works: The Copyright Issue 

Michael, Wittig; 

Raymond S.; Smith, Sally 

153 2002 Pollock, Neal Knowledge Management in Acquisition and Program Management 

(KM in the AM and PM) 

154 2002 Snider, Keith; Barrett, Considerations in Acquisition Lessons Learned Systems Design 

Frank J.; Tenkasi, 

Ramkrishnan V. 

155 2002 Johnson, Wayne M.; The Promise and Perils of Spiral Acquisition: A Practical Approach 

Johnson, Carl O. to Evolutionary Acquisition 

156 2002 Polydys, Mary Linda Interoperability in DoD Acquisition Programs Through Enterprise 

“Architecting” 

157 2002 Linster, Bruce G.; Mullin, Auctions in Defense Acquisition: Theory and Experimental 

David R. Evidence 

158 2002 Siemsen, Tom Just Contracting Parties or Partners as Well? 

159 2002 Ceylan, B. Kagan; Ford, Using Options to Manage Dynamic Uncertainty in Acquisition 

David N. Projects 

160 2002 Gaudette, Kevin; Price Enumerations and Probabilistic Evaluation in Systems 

Sweeney, Kevin Acquisition 

161 2002 Agripino, Mario; A Lean Sustainment Enterprise Model for Military Systems 

Cathcart, Tim; Mathaisel, 

Dennis 

162 2002 Nichols, Camille, LTC SCIENTIFIC WARRIORS: Do They Have a Place in the 21st 

Century Army? 

163 2002 Myers, Dominique Acquisition Reform-Inside the Silver Bullet: A Comparative 

Analysis - JDAM Versus F-22 

164 2003 Sylvester, Richard K.; Conflict and Ambiguity: Implementing Evolutionary Acquisition 

Ferrara, Joseph A. 

165 2003 Sherman, J. Daniel Patriot PAC-2 Development and Deployment in the Gulf War 

166 2003 Molzahn, Wendy The CIA’s In-Q-Tel Model its Applicability 

167 2003 Gould, Jay W. Ill Program Planning of Asynchronous On-Line Courses: Design 

Complexities and Ethics 

168 2003 Bennett, John, Bunker; Managing the Development of Technology-based Courses: Success 

Ellen, Rowley, Kurt Factors from Eight Government Training Courses 

169 2003 Coleman, Richard L.; The Relationship Between Cost Growth and Schedule Growth 

Summerville, Jessica R.; 

Dameron, Megan 

170 2003 Shepherd, Bill Managing Risk in a Program Office Environment 

171 2003 Bolles, Mike Understanding Risk Management in the DoD 

172 2003 Roberts, Barney; Smith, Risk-Based Decision Support Techniques for Programs and Projects 

Clayton; Frost, David 

173 2003 Conrow, Edmund H. Development of Risk Management Defense Extensions to the PMI 

Project Management Body of Knowledge 


Cost, Schedule, 

Performance 
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Cost, Schedule, 

Performance 
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International 
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System of Systems 

Laws, Policies, and 
Regulations 

Technology 

Technology 

Evolutionary 

Acquisition 

Technology 

Contracting 

Organizational 

Behavior 

Risks 

Contracting 

Cost, Schedule, 
Performance 

Education and Training 

Acquisition Reform 

Evolutionary 
Acquisition 
Technology 
Technology 
Education and Training 

Education and Training 

Cost, Schedule, 
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Risk 

Risk 

Risk 
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174 

2003 

Garvey, Paul R.; Cho, 
Chien-Ching 

An Index to Measure a System’s Performance Risk 

Risk 

175 

2003 

Mumyak, George; 
Leggieri, Michael J.; 
Roberts, Welford C. 

The Risk Assessment Process Used in the Army’s Health Hazard 
Assessment Program 

Risk 

176 

2003 

Waller, Robert L. 

The Use of Offsets in Foreign Military Sales 

Cost, Schedule, 
Performance 

177 

2003 

Graham, Robert 

The Transformation of Contract Incentive Structures 

Contracting 

178 

2003 

King, David R.; 
Driessnack, John D. 

Investigating the Integration of Acquired Firms in High-Technology 
Industries Implications for Industrial Policy 

Technology 

179 

2003 

Proctor, Michael D.; 
Posey-Macalintal, Amy; 
Kulonda, Dennis 

Why the “T” in Smart: A Constructive Synergy 

Education and Training 

180 

2003 

Sherman, Daniel J. 

Lessons Learned from the Early Stages of Development of the 
Guardrail Common Sensor for the Radical Reduction of Cycle Time 

Cost, Schedule, 
Performance 

181 

2004 

Morell, Jonathan A. 

Evaluating the Impact of Electronic Business Systems: Lessons 
Learned from Three Cases at the Defense Logistics Agency 

Logistics 

182 

2004 

Alford, Lionel D. 

How Compensation in Test and Evaluation Affects Aircraft 
Acquisition 

RDT&E 

183 

2004 

Rogers. Edward W.; 
Birmingham, Robert P. 

A Ten-Year Review of the Vision for Transforming the Defense 
Acquisition System 

Acquisition Reform 

184 

2004 

Driessnack, John D.; 

King, David R. 

An Initial Look at Technology and Institutions on Defense Industry 
Consolidation 

Technology 

185 

2004 

Sipple, Vince; White, 
Edward; Greiner, Michael 
A. 

Brown, Steve; Miller, 
Scott; Schvaneveldt, Kent 

Surveying Cost Growth 

Cost, Schedule, 
Performance 

186 

2004 

Recommendations on Coaching Strategies for Implementing Lean 

Lean Implementation 

187 

2004 

Srinivasan, Mandyam; 
Jones, Darren; Miller, 

Alex 

Applying Theory of Constraints Principles and Lean Thinking at the 
Marine Corps Maintenance Center 

Lean Implementation 

188 

2004 

Rebentisch, Eric; Jobo, 
Ronald 

Lean Now - Using a Research Community to Understand Change in 
the Acquisition Enterprise 

Lean Implementation 

189 

2004 

Veech, David S. 

A Person-Centered Approach to Sustaining a Lean Environment - 
Job Design for Self-Efficacy 

Lean Implementation 

190 

2004 

Joyce, Michael; 

Schechter, Bettina 

The Lean Enterprise - A Management Philosophy at Lockheed 

Martin 

Lean Implementation 

191 

2004 

Ott, David D.; Davis, 
James B 

T-38C Transition to Lean 

Lean Implementation 

192 

2004 

Blair, Bob; McKenzie, 

Jon W. 

Raytheon- New Challenges, New Solutions, and Documented 

Results 

Lean Implementation 

193 

2004 

Searcy, Dewayne; 

Greene, Brad; Reeve, 
James 

Taming the Aerospace Supply Chain - A Case Study in 
Organizational Integration 

Logistics 

194 

2005 

Friar, Allen 

Why Training for Service Contract Management is Mission 

Essential 

Contracting 

195 

196 

2005 

2005 

Schilling, Melissa A.; 
Paparone, Christopher 
Edison, Tom 

Modularity an Application of General Systems Theory to Military 
Force Development 

Social Networking Analysis: One of the First Steps in Net-centric 
Operations 

Technology 

Technology 

197 

2005 

Giammarco, Kristin; 
Carlomusto, Michael; 
Lock, J.D. 

Development and Analysis of Integrated C4ISR Architectures 

SE 

198 

2005 

Eiband, David 

Innovative Procurement Strategies 

RDT&E 

199 

2005 

Dillard, John T. 

Toward Centralized Control of Defense Acquisition Programs 

Acquisition Reform 

200 

2005 

Brown, Mary Maureen; 
Flowe, Rob 

Joint Capabilities and System-of-Systems Solutions 

Acquisition Reform 

201 

2005 

Switzer, Sandra P.; 
Stropku, Michael A. 

Effects of Defense Globalization: An Examination of Current and 
Future Command and Control Collaborations 

International 

202 

2005 

Monaco, James V.; 

White, Edward D. Ill 

Investigating Schedule Slippage 

Cost, Schedule, 
Performance 

203 

2005 

Greene, Harold; 

Mendoza, Robert 

Lessons Learned from Developing the ABCs 6.4 Solution 

System of Systems 

204 

2005 

Zenishek, Steven G.; 
Usechak, David 

Net-Centric Warfare and its Impact on Systems-of-Systems 

System of Systems 

205 

2005 

Farr, John V.; Johnson, 
William R.; Birmingham, 
Robert P. 

Multitiered Approach to Army Acquisition 

DoD Framework 
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206 2005 Hickok, John, USN (ret) Knowledge Sharing , Communities of Practice, and Learning Asset 

Integration - DAU’s Major Initiatives 

207 2005 Garcia, Jill; Dorohovich, The Truth about Building and Maintaining Successful Community 

Michael; of Practice 

208 2005 Dickover, Noel Supporting Community of Interest in a Net-Centric Investment 

Environment 

209 2005 Perino, George H. Toward More Innovative Program Management 

210 2005 Garner, James; Bundy, Tank Gun Barrel Reshaping - Concept to Implementation 

Mark; Pomey, Albert; 

Roy, Walter 

211 2005 Weber, Katherine L.; Maintaining a Viable Energy Savings Performance Contract 

Huckeby, Michael 

212 2005 Graham, Robert An Introduction to Government Technical Valuations - A 

Contracting Officer’s Perspective 

213 2005 Boudreau. Mcihael; Total Ownership Cost Considerations in Key Performance 

Naegle, Brad R. Parameters and Beyond 

214 2005 Devries, Hank J. Performance Based Logistics - Barriers and Enablers to Effective 

Implementation 

215 2005 Berkowitz, David; Gupta, Defining and Implementing Performance-Based Logistics in 

Jatinder N.D.; Simpson, Government 
James T.; McWilliams, 

Joan B. 

216 2005 Mahafza, Sherry; A Performance-Based Technology Assessment Methodology to 

Componation, Paul; Support DoD Acquisition 

Tippett, Donald 

217 2005 Gaines, Leonard T.; Service Level Agreements as Vehicles for Managing Acquisition of 

Michael, James Bret Software -Intensive Systems 

218 2005 Starks, Glenn L. Public and Private Partnerships in Support of Performance-Based 

Logistics Initiatives- Lessons Learned from Defense Logistics 
Agency Partnership 

219 2005 Friar, Edward Allen Performance-Based Service Acquisition (PBSA), A-76 and Personal 

Services - A Cautionary Note 

220 2005 Cavadias, John Contract Administration in Performance-Based Acquisition 

Environment is Serious Business 

221 2005 Barnette, Gregory L. Test and Evaluation in a Dynamic Acquisition Environment 

222 2006 Flood, Scott; Richard, An Assessment of the Lead Systems Integrator Concept as Applied 

Paul to the Future Combat System Program 

223 2006 Lechner, David; Kaiser, The Fortress and the Bazaar: Open-Source and DoD Software 

Harold 

224 2006 Cottengim, David Irreducible Truths of Software-Intensive Program Management 

225 2006 Garvey, Paul R.; Cho, An Index to Measure and Monitor a System-of-Systems 

Chien-Ching Performance Risk 

226 2006 Paparone, Christopher R.; Rubric Cubed: Are We Prisoners of Orsa-Style Decision Making 

Crupi, James A. 

227 2006 Sherman, Daniel J. Lessons Learned from the Development of the Fiber Optic Guided 

Missile (FOG-M) 

228 2006 Wissler, John B. Technology Transition: A More Complete Picture 

229 2006 Moretto, Stephen J. Technology Management Best Practices: Reducing Technology 

Identification, Evaluation, and Selection Costs 

230 2006 Andresen, Jon UID: Building the Permanent Foundation 

231 2006 Mukherjee, Mrinal K. Technical Evaluation of Military Ground Systems for Export 

Licensing: A Metric Approach 

232 2006 Reiffer, Donald J.; Providing Incentives for Spiral Developments: An Award Fee Plan 

Boehm, Barry W. 

233 2006 Sherman, Daniel J. Lessons Learned from the Development of the Joint Stand-Off 

Target Attack Radar System Common Ground Station 
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APPENDIX B. RESEARCHERS’ ARTICLE SAMPLING EOR 

THE THEMES 


Refer to Appendix A for the articles used in this sampling. 

Technology: Battle Labs: Tools and Scope (Article #32): This is a tool for the rapid insertion of 
new technology into weapons systems. 

Cost, Schedule, and Performance: The Cost and Benefits of the Earned Value Management 
Process (Article #84): This literature review talks about the how the cost, schedule, and 
performance using a work breakdown structure help in the management process. 
Additionally, it talks about the benefits of the indices. 

Contracting: The Transformation of Contract Incentive Structures (Article #177): An 

acquisition and operations contract combines an award fee and success fee to include a cost 
mitigation approach. The incentive program asks for government and contractor share to 
succeed. 

Acquisition Reform: Towards Centralized Control of Defense Acquisition Programs 

(Article #199): The new acquisition framework has more phases and added 

more reviews. 

Education & Training: Engineering Management Training: Comparing Experiential versus 
Lecture Methods of Instruction (Article #121): He claims that the experiential method of 
instructions is more effective in learning and soliciting student’s reaction. 

Risk: Understanding Risk Management in the DoD (Article #171): The current risk 

management framework does not adequately emphasize the interface between risk 
management and contract administration. 

Software: Irreducible Truths of Software-Intensive Program Management (Article #224): It 
argues that software-intensive program do not automatically make a program manager 
achieve the goals and objectives established in the acquisition program baseline. 

Logistics: Defining and Implementing Performance- Based Logistics in Government 

(Article #215): This article talks about Performance-Based Logistics (PBL)’s 

implementation, benefits, and infrastructure changes required. 

Laws, Policies, and Regulations: What Every Government Employee Should Know about Post- 
Eederal Employee Restrictions (Article #28): The article talks about three contradictory 
principal laws about post-federal employment restrictions on acquisition Government 
personnel. 

Organizational Behavior: Crisis in the Acquisition Workforce: Some Simple Solutions 
(Article #131): The author here suggests some approaches to deal with loss due to retirement 
and attrition since the military workforce is important to develop weapons systems. 

Lean Implementation: A Person-Centered Approach to Sustaining a Lean Environment—Job 
Design for Self-Efficacy (Article #189): Toyota is used to demonstrate the success of the lean 
concept. It explores relationships between corporate beliefs, employee satisfaction, and self- 
efficacy. 

International: International Cooperative Research and Development Programs (Article #85): 
A literature review on International Cooperative R&D Programs gives advice on how to 
succeed logistically and economically. 

Systems Engineering: Systems Engineering and the Joint Strike Eighter: The Elagship 
Program for Acquisition Reform (Article #117): The article shows how JSF uses Systems 
Engineering process as an acquisition reform initiative to achieve its warfighting capability 
while satisfying the needs of the Navy, Air Force, Marines, and the United Kingdom. 
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Research, Development, Test, Evaluation: General Flight Test Theory Applied to Aircraft 
Modifications (Article #94): The article talks about the use of fly-by-wire technology on the 
C-130. Small modifications need extensive testing as well. 

Management: An Assessment of Air Force Development Portfolio Management Practices 
(Article #147): Decision makers need to weigh benefits and cost against the mission needs 
for different weapons systems programs. 

Defense Industry: Consolidation of the U.S. Defense Industrial Base: Impact on Research 
Expenditures (Article #148): The article talks about the merging of the defense contractors 
and the reduction in R&D expenditures. 

Change: Changing the Focus of Business Process Redesign from Activity Flows to Information 
Flows: A Defense Acquisition Application (Article #132): This is an action research study of 
a business process redesign that supports that information flow instead of activity flow should 
the focus of the business redesign process. 

Streamline Acquisition: Commercial Best Practices and the DoD Acquisition Process 
(Article #46): The article uses the best practices of the American automobile industry 
(Chrysler) to streamline the DoD acquisition process. 

Program Management: Identifying Factors that Contribute to Program Success (Article #67): 
Good program management can prevent cost overruns, substandard performance, schedule 
delays, and also cancellations of programs. 

Modeling and Simulation: Building a Business Case for Modeling and Simulation 
(Article #111): It provides a framework for program managers within DoD to know when to 
apply M&S to programs. 

System of Systems: Net-Centric Warfare and its Impact on Systems-of-Systems (Article #204): 
Air Force Distributed Common Ground System Block 10.2 is fielding systems of systems 
net-centric, service-oriented architecture for intelligence, surveillance, and reconnaissance. 

DoD Framework: DoD's 5000 Documents: Evolution and Change in Defense Acquisition 
Policy (Article #37): This article presents a historical progression of the DoD 5000 series and 
documents. 

Decision Making: Using Data Types and Scales for Analysis and Decision Making 
(Article #11): It exposes data scales and numerical manipulations to help decision makers 
choose among different alternatives and make decisions on the allocation of scarce resources. 

Procurement: Purchasing Performance: A public Versus Private Sector Comparison of 

Commodity Buying (Article #98): The article compares DoD and commercial spending and 
acquisition. Careful research is needed to benefit from the purchasing reform. 

Commercial Industry: Assessing Industrial Capabilities for Carbon Tiber Production 
(Article #96): Commercial industries play an important role in the production and 
manufacturing of defense aerospace systems. 

Evolutionary Acquisition: Conflict and Ambiguity Implementing Evolutionary Acquisition 
(Article #164): Evolutionary acquisition is the new preferred approach in acquiring defense 
systems as the Secretary of Defense announced in 2000. This article features the struggles 
and challenges of policy implementation. 

IPPD/IPT: The Phoenix Rises (Article #62): This article shows how a program went from 
possible extinction to a respective model based on the ability to reduce cost and schedule yet 
still come out ahead through the use of integrated product teams (IPT). 
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APPENDIX C. ELDER’S THESIS TOP CONTRIBUTORSso 


Elder’s Top Contributors 


Authors 

Contributions 

Christensen, David, Dr. 

6* 

Nissen, Mark E., Dr. 

5 

Washington, William 

5 

Alford, Eionel D., Et Col, USAE 

4 

Arora, Ashish 

3 

Besselman, Joseph, Maj, USAE 

3 

Driessnack, John D., Et Col, USAE 

3 

Earkey, Patrick 

3 

Einster, Bruce G. 

3 

Pollock, Neal 

3 

Snider, Keith E., Dr. 

3 

Templin, Carl, Dr. 

3 


*Note: This includes an issue introduction. 
Reproduced from Elder’s thesis (2005). 


Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, p. 27. 
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APPENDIX D. ELDER’S TABLE EOR AUTHORS 

PER ARTICLE51 


Table 11. Authors per Article 

One Two Three Four 


Year Mean Author Authors Authors Authors 


1 

1.16667 

83.3% 

16.7% 

0.0% 

0.0% 

2 

1.25 

72.7% 

18.2% 

9.1% 

0.0% 

3 

1.18182 

81.8% 

18.2% 

0.0% 

0.0% 

4 

1.76 

60.0% 

8.0% 

28.0% 

4.0% 

5 


57.1% 

23.8% 

14.3% 

4.8% 

6 

1.65217 

56.5% 

26.1% 

13.0% 

4.3% 

7 

1.68182 

68.2% 

9.1% 

9.1% 

13.6% 

8 

1.28571 

78.6% 

14.3% 

7.1% 

0.0% 

9 

2.21053 

31.6% 

26.3% 

31.6% 

10.5% 

10 

1.76471 

52.9% 

17.6% 

29.4% 

0.0% 

11 

2.15385 

23.1% 

38.5% 

38.5% 

0.0% 


Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, p. 28. 
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Frequencies 


APPENDIX E. ELDER’S SUBJECT (THEME) CATEGORIES52 


Risk Management 
Buyer/Seller Relationships 
Performance and Measurement 
Organizational Issues 
Research and Development 
Industry Issues 
Contract Management 
Policy and Regulation 
Program Fielding and Implementation 
Analysis and Decision Making 
Cost and Schedule 
Interoperability 
Management and Organizational Behavior 
Acquisition Strategy 
Reform Initiatives 

0 5 10 15 20 25 30 


Subject Categories 



^2 Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, p. 19. 
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APPENDIX F. ELDER’S FREQUENCY AND PROPORTION OF 
SUBJECT CATEGORY PER PERIOD53 



1 

2 

3 



1 

2 

3 

Reform Initiatives 

9 

11 

7 

27 

Reform Initiatives 

13.8% 

16.9% 

11.1% 

Acquisition Strategy 

6 

9 

6 

21 

Acquisition Strategy 

9.2% 

13.8% 

9.5% 

Management and 





Management and 




Organizational Behavior 

3 

7 

10 

20 

Organizational Behavior 

4.6% 

10.8% 

15.9% 

Interoperability 

4 

6 

10 

20 

Interoperability 

6.2% 

9.2% 

15.9% 

Cost and Schedule 

7 

7 

4 

18 

Cost and Schedule 

10.8% 

10.8% 

6.3% 

Analysis and Decision Making 

6 

6 

1 

13 

Analysis and Decision Making 

9.2% 

9.2% 

1.6% 

Program Fielding and 





Program Fielding and 




Implementation 

4 

5 

4 

13 

Implementation 

6.2% 

7.7% 

6.3% 

Policy and Regulation 

9 

2 

0 

11 

Policy and Regulation 

13.8% 

3.1% 

0.0% 

Contract Management 

5 

2 

3 

10 

Contract Management 

7.7% 

3.1% 

4.8% 

Industry Issues 

2 

1 

6 

9 

Industry Issues 

3.1% 

1.5% 

9.5% 

Research and Development 

3 

2 

2 

7 

Research and Development 

4.6% 

3.1% 

3.2% 

Organizational Issues 

2 

3 

1 

6 

Organizational Issues 

3.1% 

4.6% 

1.6% 

Performance and 





Performance and 




Measurement 

2 

2 

2 

6 

Measurement 

3.1% 

3.1% 

3.2% 

Buyen'Seller Relationships 

2 

2 

2 

6 

Buyep'Seller Relationships 

3.1% 

3.1% 

3.2% 

Risk Management 

1 

0 

5 

6 

Risk Management 

1.5% 

0.0% 

7.9% 
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65 

63 







^3 Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, p. 20. 
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APPENDIX G. ELDER’S PROPORTION OE 
CONTRIBUTING INSTITUTIONS54 



1 

2 

3 

Civilian Universities 

14.1% 

24.0% 

28.6% 

DSMC/DAU 

19.7% 

12.0% 

10.4% 

Civilian Research Firm.' 
Defense Contractor 

9.9% 

12.0% 

19.5% 

Other USAF 

1.4% 

16.0% 

10.4% 

Other USA 

12.7% 

6.7% 

7.8% 

ICAF 

12.7% 

2.7% 

2.6% 

NPS 

4.2% 

8.0% 

3.9% 

AFIT 

9.9% 

2.7% 

2.6% 

SECDEF 

5.6% 

5.3% 

1.3% 

USAFA 

4.2% 

1.3% 

3.9% 

Other USN 

1.4% 

5.3% 

2.6% 

Other DoD 

2.8% 

0.0% 

1.3% 

National Defense University 

1.4% 

0.0% 

1.3% 

ACSC 

0.0% 

1.3% 

1.3% 

Air War College 

0.0% 

1.3% 

0.0% 

Other Gov 

0.0% 

1.3% 

0.0% 

USMA 

0.0% 

0.0% 

1.3% 

Naval War College 

0.0% 

0.0% 

1.3% 


^4 Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, p. 29. 
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APPENDIX H. ELDER’S PROPORTION OE 
PRACTITIONERS VS. ACADEMICS55 


Table 14. Pi oportiou of Practitioners vs. Academics 


Contributor Type 

1 

2 

3 

Practitioners 

66.2% 

53.3% 

55.8% 

Academics 

33.8% 

46.7% 

44.2% 


Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, p. 30. 
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APPENDIX I. ELDER’S PROPORTION OE 
CIVILIAN VS. GOVERNMENT ORGANIZATIONS56 


Table 13. Proportion of Civilian vs. Govei iiineiit Orgaiiiziitioiis 


Contributor Type 

1 

2 

3 

Civilian Organizations 

23.9% 

36.0% 

48.1% 

Military Organizations 

76.1% 

64.0% 

51.9% 


Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, p. 30. 
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APPENDIX J. ANOVA TEST EOR THE AUTHORS PER 
ARTICLE (233 ARTICLES) 


Anova: Single Factor 


SUMMARY 


Groups 

Count 

Sum 

Average 

Variance 

1994-1997 

64 

92 

1.4375 

0.59921 

1998-2000 

66 

109 

1.65152 

0.99977 

2001-2003 

50 

90 

1.8 

0.89796 

2004-2006 

53 

94 

1.77358 

0.67852 

ANOVA 


Source of Variation 

SS 

df 

MS 

F 

P-value 

F crit 

Between Groups 

4.82333 

3 

1.60778 

2.02277 

0.1115 

2.64402 

Within Groups 

182.018 

229 

0.79484 




Total 

186.841 

232 
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APPENDIX K. CHI-SQUARED TEST EOR AUTHORS PER 
ARTICLE (233 ARTICLES) 


ACTUAL FRE 

QUENCY 

No. of 

Authors/Article 

One 

Author 

Two 

Authors 

Three + 
Authors 

Period 1 (1994-1997) 

46 

9 

9 

Period 2 (1998-2000) 

41 

13 

12 

Period 3 (2001-2003) 

26 

10 

14 

Period 4 (2004-2006) 

23 

21 

9 


136 

53 

44 

EXPECTED FR] 

EQUENCY 

No. of 

Authors/Article 

One 

Author 

Two 

Authors 

Three + 
Authors 

Period 1 (1994-1997) 

37 

15 

12 

Period 2 (1998-2000) 

39 

15 

12 

Period 3 (2001-2003) 

29 

11 

9 

Period 4 (2004-2006) 

31 

12 

10 


136 

53 

43 
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APPENDIX L. ANOVA TEST EOR THE NUMBER OE AUTHORS 
PER ARTICLE (193 ARTICLES TO COMPARE WITH 

ELDER’S THESIS) 


Anova: Single Factor 


SUMMARY 


Groups 

Count 

Sum 

Average 

Variance 

Column 1 

64 

92 

1.4375 

0.59921 

Column 2 

66 

109 

1.65152 

0.99977 

Column 3 

63 

117 

1.85714 

0.8341 

ANOVA 


Source of Variation 

SS 

df 

MS 

F 

P-value 

F crit 

Between Groups 

5.59232 

2 

2.79616 

3.43977 

0.03408 

3.04347 

Within Groups 

154.449 

190 

0.81289 




Total 

160.041 

192 
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APPENDIX M. ELDER’S CATEGORIES57 


Article Categories _ Elder (2005) 


Type 

Period 1 

Period 2 

Period 3 

Normative 

44.6% 

46.2% 

33.3% 

Literature Review 

3.1% 

1.5% 

3.2% 

Exploratory 

16.9% 

15.4% 

19.0% 

Methodology 

32.3% 

33.8% 

38.1% 

Hypothesis 

3.1% 

3.1% 

6.3% 


Research Designs _ Elder (2005) 


Type 

Period 1 

Period 2 

Period 3 

Surveys 

1.5% 

10.8% 

4.8% 

Case Study/ies 

13.8% 

13.8% 

22.2% 

Interviews 

6.2% 

3.1% 

6.3% 

Archival Study 

29.2% 

21.5% 

22.2% 

Experiment 

0.0% 

3.1% 

3.2% 

Simulation 

N/A 

N/A 

N/A 

Topic Presentation 

49.2% 

47.7% 

39.7% 

Mathematical Model 

0.0% 

0.0% 

1.6% 





N/A = Not Applicable. 





Article Categories _ Elder (2005) 


Type 

Period 1 

Period 2 

Period 3 

Anecdotal Evidence 

26.2% 

20.0% 

22.2% 

Statistical 

6.2% 

21.5% 

20.6% 

Content Analysis 

15.4% 

7.7% 

11.1% 

Comparison Analysis 

7.7% 

9.2% 

4.8% 

No Analysis 

44.6% 

41.5% 

41.3% 


Mitchell Elder, Capt, USAF, “An Eleven Year Retrospective of the Acquisition Review Journal,” 
Master’s thesis. Air Force Institute of Technology, March 2005, pp. 23-24, 26. 
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